824 Commits

Include renames

Author	SHA1	Message	Date
Ruben Fiszelandwindmill-internal-app[bot]	cfcfe298dd	feat(ai-chat): make reusable skills ai_skill resources you select per workspace (#10914) ... * feat(ai-chat): make reusable skills ai_skill resources you select per workspace * chore: pin the ee ref to the skill telemetry counters * fix: address review findings on skill authoring, import and migration * fix: enforce skill selection in read_skill and stop imports clobbering resources * feat: carry format_extension from the hub into synced resource types * fix: let an edit set or clear a resource type's format_extension * fix: regenerate the sqlx cache and close the review round findings * fix: close the round-2 findings on folder ACLs, cached sync and truncation * refactor: make the skills migration non-destructive and use design-system inputs * fix: close the round-4 findings on folder owners, startup sync and truncation * fix: clear obsolete extensions, guard folder owners, and report skipped skills * fix: honor explicit-null extensions and report same-type migration conflicts * fix: scope skill actions to the committed workspace and paginate the listing * fix: keep the drawer scoped to the live workspace and surface truncation * fix: discard a skills refresh for a workspace the chat has left * chore: update ee-repo-ref to 6efe7a73c745c2e1377a34498523c00d89010a3d This commit updates the EE repository reference after PR #764 was merged in windmill-ee-private. Previous ee-repo-ref: 55998c142bc72edd08532748af1974b16035658d New ee-repo-ref: 6efe7a73c745c2e1377a34498523c00d89010a3d Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-09-01 12:51:27 +00:00
3 people	aa4a6ffd66	fix: track outstanding service log files on the rows themselves (#10894) ... * fix: track outstanding service log files on the rows themselves Adds `log_file.indexed_at` so the service log ingest can read outstanding rows instead of walking a cursor over `log_ts`. A row registered after the pass had gone by its minute was skipped for good, and no ordering fixes that — an arrival sequence fails the same way, since a row can take a lower value and commit after a higher one has moved the cursor past it. The migration marks existing rows with a sentinel; the first pass returns the ones the old cursor had not reached to the queue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPAP96jJNYpPQ8bpxZcU1C * [ee] refactor: drop the claim/confirm phase from the service log ingest queue Two states are enough: a row is outstanding or it is marked. The migration no longer creates the index for the claim sentinel, and the sqlx cache loses the two queries the event-time cursor used. * [ee] fix: make re-indexing a service log file idempotent Corrects the `init_last_log_file_sent` note: a rewritten row keeps the `indexed_at` it had, so one the indexers already took is not offered again. * [ee] fix: let a rebuild take the rows it covered out of the ingest queue Adds the query that releases them; the index layout stays v4. * [ee] fix: index the lookup a rebuild releases rows by A rebuild takes rows out of the queue by the file it read out of the store, which is the one lookup that arrives without a `log_ts`. The primary key is `(hostname, log_ts)`, so nothing covered it and each batch scanned every outstanding row — worst in exactly the state a rebuild follows. Verified at 50k outstanding rows: sequential scan becomes an index scan. Also records `log_file.indexed_at` in the schema reference. * [ee] fix: treat a state handed back without its line count as behind * [ee] fix: give the converted state a line count * [ee] fix: keep the converted cursor from being rewound by the rebuild * [ee] fix: inherit the legacy cursor from one source, not field by field * [ee] fix: count a file's lines against the buffer before reading it * [ee] fix: bound the row buffer on what it holds, not on reported counts * [ee] fix: settle the upgrade from the store rather than from event time * [ee] docs: describe the conversion's second half as it now works * [ee] refactor: settle the upgrade with one rebuild instead of reconciling The migration records existing rows as done rather than marking them with a sentinel: the indexer puts back what the old cursor had not reached on its first pass, which is the only place that cursor's position is known. * [ee] fix: repair the rows the old cursor skipped instead of recording them as done The migration marks pre-existing rows with a sentinel again, so the indexer can tell them from rows registered since and put the window's worth back on the queue. * [ee] fix: keep a source file whole in one partition * [ee] revert the file-atomic partition change * [ee] fix: dedupe the public reads, and repair an index without a cursor * [ee] fix: repair an index whose cursor is gone, and keep what the repair found * [ee] fix: seed a pass from both axes of what a rebuild recovered * [ee] fix: settle the cursor on what the store holds, not on what was read * [ee] fix: an empty rebuild must not claim ground it has not covered * [ee] test: pin the cursor a rebuild settles on * chore: update ee-repo-ref to bc0c7051585194474078b6c1941a3fb73893d9e5 This commit updates the EE repository reference after PR #755 was merged in windmill-ee-private. Previous ee-repo-ref: 328f5a90afeae9c683bf3294f0d9eb293a3e1a92 New ee-repo-ref: bc0c7051585194474078b6c1941a3fb73893d9e5 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-31 14:06:29 +02:00
3 people	d91ee4614a	feat: day-partition the service log index and expire whole chunks (#10893) ... * feat: day-partition the service log index and expire whole chunks The service log index becomes one tantivy index per UTC day. The substance is in windmill-ee-private#753; this side carries the EE ref and moves the log indexer writer instead of cloning it, because sealing a chunk takes sole ownership of its tantivy writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: do not adopt the superseded watermark after an explicit index clear A clear asks for the retention window to be read again, and a watermark says it already has been — and the v3 copy in object storage is kept for rollback, so it outlives the local one the clear removes. Both copies of that watermark are now read and the newer wins, for the same reason the v4 one is taken from the store when it is ahead: a replica that lost the lock keeps a local file frozen where it stopped while the store went on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: delete a day's raw files at its checkpoint, and rebuild whole days Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: make an interrupted rebuild detectable, and pin the rebuild floor Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: keep the rebuild marker in the object store, not on local disk Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: two more routes to a partial index being accepted as complete Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: trust a local chunk only when the tracker vouches for it Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * chore: condense the stale-chunk guard's doc to the four-line limit Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * chore: update ee-repo-ref to 17ef439b087b400889ff19109be9d2c810142278 This commit updates the EE repository reference after PR #753 was merged in windmill-ee-private. Previous ee-repo-ref: 3e79901b4742906d2285dd943e24fac0f735f199 New ee-repo-ref: 17ef439b087b400889ff19109be9d2c810142278 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-30 07:16:09 +02:00
3 people	815de49e23	feat: make the service log retention period an instance setting (#10889) ... * feat: make the service log retention period an instance setting Service log retention was a hardcoded 14 days with no override, unlike job retention. It becomes the `service_log_retention_secs` global setting (env `SERVICE_LOG_RETENTION_SECS`, default unchanged at 14 days), reloaded on change like the other retention settings. The constant becomes `DEFAULT_SERVICE_LOG_RETENTION_SECS` and every reader goes through `service_log_retention_secs()`, so the `log_file` sweep, the object-storage orphan scan, the columnar store's compaction and pruning, the retrieval clamp and the search index's trim window all follow the configured value. Loaded outside `initial_load`'s `server_mode` guard: a dedicated indexer trims the search index to a window derived from this value and is not a server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: never let a non-positive service log retention expire every log Every service log cutoff is `now - retention`, so a `0` or negative window puts the cutoff at or after `now` and the next sweep reads the whole history as expired — deleting the `log_file` rows and their object-storage files irreversibly. `0` is reachable two ways now that the window is configurable: it is what an operator types by analogy with the job retention period sitting directly above it, where `0` does mean keep forever; and `SecondsInput` writes a `0` into a field that was merely focused, so saving the Jobs panel is enough. Service logs always have a window, so clamp an unusable value back to the default in the accessor every reader already goes through. The upper bound is where `chrono::Duration::seconds` panics, which would abort the sweep that reads it. The settings field rejects a non-positive value rather than silently correcting it, and its description now names the database rows too — they are swept on every instance, including one with no object storage configured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: address review findings on the service log retention setting - Bound the monitor's `log_file` sweep. Every process rotates a log file a minute, so lowering the retention can make one ordinary setting change expire millions of rows; the unbounded `DELETE ... RETURNING` materialized all of them, and their deletion futures, in a single tick. Batched like the settings-page cleanup on the same table. - Make the retention atomic private and give it one writer, so a value that would expire every service log cannot reach a cutoff by any path, and say so in the log when one is rejected rather than falling back silently. - Cap the retention at a century. The previous ceiling only bounded `TimeDelta` construction, while consumers compute `now - retention`, which panics past year 262143, and build a Postgres interval that overflows well before the old cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: cap an oversized service log retention instead of shortening it The two unusable directions were landing on the same fallback, so configuring a retention above the ceiling silently produced 14 days — deleting logs the operator had asked to keep for longer. Too large now caps at the maximum, which preserves that intent; only a non-positive value, which would expire everything and has no upward reading, falls back to the default. Also bound the `log_file` drain to ten batches per pass: `monitor_db` runs under a 600s timeout that cancels every maintenance future in the same `join!` and reports a critical error, so a backlog large enough to need batching has to drain across ticks, the way the neighbouring sweeps already do. The settings field carries the upper bound too, and the superseded query's offline entry is dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: route the new log-file registration cutoff through the retention accessor `send_log_files_to_object_store` arrived on main while this branch was open and reads the retention directly. The atomic behind it is private now, so it goes through the accessor like every other consumer — which also means the cutoff it uses to skip registering already-expired files follows the configured retention rather than a fixed two weeks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: say why every mode loads the service log retention setting A worker registers its rotated log files against the retention cutoff, so the comment naming only the indexer no longer covers why the setting sits outside the `server_mode` guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: file service log retention under Monitoring, not Jobs Service logs are the Windmill processes' own logs — every process rotates and registers its own, no job involved — so the Jobs panel was grouping by the shape of the widget rather than by the subject. It sits under Monitoring now, beside the Indexer panel that holds the other service-log window. Its own section rather than inside that panel: the panel is badged EE, while this governs the database sweep that runs on every instance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * chore: update ee-repo-ref to a6e3533b26195918a17fea58646f71d2bbcde288 This commit updates the EE repository reference after PR #752 was merged in windmill-ee-private. Previous ee-repo-ref: 1d93da24bd166b9a5a5cc204034a1d35ffc88474 New ee-repo-ref: a6e3533b26195918a17fea58646f71d2bbcde288 Automated by sync-ee-ref workflow. * feat: say on the service logs page where the logs actually are The retention number alone does not tell an operator what it governs, and the answer differs by instance. Two states are worth calling out because they are the ones where retention does not mean what it looks like: Without instance object storage, each process keeps its files on its own disk. The page lists what every host wrote, since the rows are in the shared database, but can only open the files of the replica serving the request, and a host's files go with it when it is replaced. With object storage but "Delete logs from s3 periodically" off — the backend default, since uploads are gated on a store existing while deletions are gated on that toggle — expiring a log removes the row and the local file and leaves the uploaded copy behind for good. The retention field itself now names every copy it covers and says that full-text search reaches back at most that far, and less when the indexer's own window is shorter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: describe raw log files as the transient copy they became Retiring the raw files landed while this was being written: the indexer now deletes each one as soon as it is ingested, and the log viewer rebuilds a file from the columnar store once the raw copy is gone. So the durable copy is the store, and warning that an uploaded file is kept forever when periodic s3 deletion is off only holds where no indexer runs to ingest it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * chore: point ee-repo-ref at the EE compile fix EE main does not build on its own: extracting the index-window expression and adding a fourth copy of it landed in separate PRs that never conflicted textually. windmill-ee-private#756 is the one-line fix; this pins it so CI has a tree that compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-29 19:39:14 +02:00
Ruben FiszelandClaude Opus 5	c8172480b0	fix: register every rotated service log file exactly once (#10891) ... * fix: register every rotated service log file exactly once Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QTxWg4Sx57UodA9RpFMJm * chore: refresh sqlx cache for the log_file watermark query Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QTxWg4Sx57UodA9RpFMJm * fix: skip service log files past the retention cutoff on catch-up Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QTxWg4Sx57UodA9RpFMJm * refactor: name the shutdown flush for what it does and scope its doc claims Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QTxWg4Sx57UodA9RpFMJm --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-29 11:53:21 +02:00
3 people	7c1a785f75	feat: serve service log retrieval from a columnar parquet store (#10886) ... * feat: always write service log files as json so they index structured Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ijGGPCFkYhVzisFexAHYx * feat: serve service log retrieval from a columnar parquet store Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ijGGPCFkYhVzisFexAHYx * feat: shrink the service log index to the per-host count it still serves Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ijGGPCFkYhVzisFexAHYx * fix: reclaim the superseded service log index on upgrade Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ijGGPCFkYhVzisFexAHYx * fix: address review findings in the service log store Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ijGGPCFkYhVzisFexAHYx * chore: update ee-repo-ref to ad9e899dfd2ee4e3d18ecf06d016f821968c5a83 This commit updates the EE repository reference after PR #751 was merged in windmill-ee-private. Previous ee-repo-ref: 6ad4064f9d58d83612b42b4ec870384994d64bcb New ee-repo-ref: ad9e899dfd2ee4e3d18ecf06d016f821968c5a83 Automated by sync-ee-ref workflow. * fix: address review nits on the service log store Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ijGGPCFkYhVzisFexAHYx --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-29 09:51:59 +02:00
4 people	b72ccc3593	fix: key build artifact caches on a runnable's inline modules (#10819) ... * fix: key build artifact caches on a runnable's inline modules Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: seal the cache-key base and skip prebundling multi-file bun scripts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: tighten cache-key invariant comments and name the retained-artifact residual Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: version the build artifact keyspace so pre-fix artifacts are abandoned Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: namespace the artifact cache by keyspace version instead of the hash preimage Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: namespace module-bearing artifacts instead of versioning the whole keyspace Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: pin the cache-name base seal and name the retained-artifact residual Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: bump ee ref for agent-worker module resolution fix Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: align agent-worker module resolution with the worker for previews by hash Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: drop calculate_hash imports left unused by artifact_cache_name Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 2d6c66b32f20d9605c6a677727473ab66fcc8a87 This commit updates the EE repository reference after PR #743 was merged in windmill-ee-private. Previous ee-repo-ref: efce983cae3d53175bbb286a10205a2a360c2a9e New ee-repo-ref: 2d6c66b32f20d9605c6a677727473ab66fcc8a87 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-08-28 16:40:22 +02:00
hugocasaandClaude Opus 5	ffdf17ef8d	fix: force HTTP router rebuild on trigger-change notification (#10849) ... * fix: force HTTP router rebuild on trigger-change notification Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: coalesce http trigger change events into one forced rebuild Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: retry the coalesced http router rebuild when it fails Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: mark http routers stale when a forced rebuild fails Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep the router invalidation across an in-flight rebuild Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-26 08:23:25 +02:00
hugocasaandClaude Opus 5	2906504125	feat: add instance setting to mute zombie job restart alerts (#10813) ... * feat: add instance setting to opt out of zombie job restart alerts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: preserve explicit false for default-on boolean instance settings Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor: invert zombie restart alert setting to a mute flag Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-24 22:36:21 +02:00
Ruben FiszelandClaude Opus 5	4b406e37c0	fix: size the ephemeral job token to the job timeout it must serve (#10804) ... * fix: size job token to the premium cloud job timeout Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: give the job token setup headroom and drop dead MAX_TIMEOUT_DURATION Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cap job token setup slack so self-hosted tokens stay at 7d Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 09:30:08 +00:00
Ruben FiszelandClaude Opus 5	3c8e4b43fd	fix: resolve a script path to its new version as soon as the lock lands (#10794) ... * fix: resolve a script path to its new version as soon as the lock lands Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011H5ygpzQHkPeYsjiP9GzBy * fix: tell MCP script deploy callers to stop polling on a lock error Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011H5ygpzQHkPeYsjiP9GzBy --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 11:54:51 +02:00
3 people	1f59841a67	feat: add WM_ROOT_WORKSPACE, the closest dev or prod workspace of a job (#10776) ... * feat: add WM_ROOT_WORKSPACE, the closest dev or prod workspace of a job Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HBMJwogo6jJ1P55uvB3YpF * fix: do not cache a failed root-workspace lookup, and sweep on fork create Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HBMJwogo6jJ1P55uvB3YpF * fix: shorten the agent-worker root-workspace TTL and pin the sweep wiring Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HBMJwogo6jJ1P55uvB3YpF * chore: update ee-repo-ref to a2fa58e5301d3865dd06ad73519e20ba7a5af0f0 This commit updates the EE repository reference after PR #736 was merged in windmill-ee-private. Previous ee-repo-ref: 07a9d26a79a403ae27c48abd508a6699f2c87c49 New ee-repo-ref: a2fa58e5301d3865dd06ad73519e20ba7a5af0f0 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-20 15:40:22 +02:00
Ruben FiszelandClaude Opus 5	de90e44650	docs: document DATABASE_URL_FILE in the env var help (#10773) ... Claude-Session: https://claude.ai/code/session_0125f1Vwj7pR9oY8NCxLXwtW Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 12:35:35 +02:00
Ruben FiszelandClaude Opus 5	0258f3f81b	perf: unblock workers before the API router is built (#10711) ... Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 16:43:51 +02:00
Ruben FiszelandClaude Opus 5	c9ddddda1b	log the settings a failed read left unapplied (#10709) ... Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-14 21:15:33 +02:00
4 people	53eb94659b	feat(telemetry): extend feature-usage tracking beyond AI features (#10681) ... * feat(telemetry): extend feature-usage tracking to long-tail features Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: describe telemetry as product feature usage rather than AI usage Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(telemetry): trim disclosure copy and drop unused pick origin Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(telemetry): count trigger fires per run and key hub picks from hub data Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(telemetry): slugify hub keys and order both writers' upserts Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(telemetry): key native trigger adoption by service so it matches fires Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref for native trigger adoption fix Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(telemetry): move feature-usage collection into the ee crate * docs: point feature-telemetry at the moved registry and rust writer * docs: correct the trigger-fire gate comment to match measured step counts * docs: put the private-build caveat on the verification step * chore: update ee-repo-ref to f079db9e7962a413b349c4ff8036080894f30771 This commit updates the EE repository reference after PR #725 was merged in windmill-ee-private. Previous ee-repo-ref: 055adb80416f9339c9a28ae7fbaeadad30d74959 New ee-repo-ref: f079db9e7962a413b349c4ff8036080894f30771 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-14 18:50:38 +02:00
Ruben FiszelandClaude Opus 5	30f5d2e766	perf: declare a settings pass instead of reading one setting at a time (#10698) ... * perf: read global_settings once per settings-load pass `initial_load` reads several dozen settings back to back, one `SELECT value FROM global_settings WHERE name = $1` each: 50 serialized round trips before a worker is ready, 32 before a server is. On localhost that is ~20ms and invisible; against a real database it is 50x the RTT per process start, which `EXIT_AFTER_N_JOBS` turns into a per-job cost. `with_global_settings_snapshot` reads the whole table (12 rows on a typical instance) into a tokio task-local, and `load_value_from_global_settings` serves from it. Scoping it to the task is what keeps the single-setting reload paths correct: a `notify_global_setting_change` event for one key runs outside any scope and still reads the database, so a live settings change reaches a running worker as before. Agent workers hold an HTTP connection with no snapshot to take and are unchanged. `load_smtp_config` and `reload_custom_tags_setting` had their own inline copies of the same query; they go through the shared loader so they land in the snapshot too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the snapshot contract on the reader and the query `load_value_from_global_settings` is called from ~10 crates and one of them writes a setting then immediately re-reads it through `reload_custom_tags_setting`; say on the function itself that a scope, when one is installed, serves the read and leaves `db` unused. The query comment claimed the table is a handful of rows. It is not bounded that way: `workspace_dependencies_map_rebuilt:<workspace_id>` adds a row per workspace and never removes it. Those dynamically named rows are also why the snapshot fetches the whole table instead of the wanted names, so state that as the reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound the settings snapshot and keep it out of two reads Three review findings, all real: The snapshot fetched the whole table, which is not bounded by the settings that exist: `workspace_dependencies_map_rebuilt:<workspace_id>` adds a row per workspace with no cleanup path, and no settings pass reads one. It now fetches only statically named rows, and reads of a `<prefix>:<id>` name skip the snapshot and go to the database. Correctness does not rest on that naming convention — a colon-free dynamic name would simply be in the snapshot and still answered correctly — only the bound does. A snapshot query that failed inside an enclosing snapshot awaited the body bare, so its reads were served by the outer snapshot rather than falling through as documented. The task-local carries an explicit bypass state and the failure path scopes it. `reload_jwt_secret_setting` decided whether to generate-and-upsert the JWT secret from a snapshot-served read, so a replica booting alongside another could overwrite the secret it had just generated and invalidate its tokens. That read goes through the new `load_value_from_global_settings_fresh`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the snapshot query on the primary-key index `name NOT LIKE '%:%'` bounded the rows returned but not the work: a leading wildcard cannot use the index, so Postgres read every row anyway. Against 50k dynamically named rows it plans as a seq scan of 516 buffers whether or not seqscans are enabled — and worker connections disable them, so the plan was one the query shape forbade rather than one the planner chose. `name = ANY($1)` over an explicit list plans as a bitmap index scan, 7 buffers, bounded by the listed names rather than by table size. That list is also exactly the set the snapshot may answer from, so a name outside it falls through to the database instead of reading as unset: listing a setting is a performance choice, never a correctness one, which is what keeps the list safe to maintain by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: declare a settings pass instead of reading one setting at a time Replaces the prefetch-list snapshot with a pass the call sites build themselves. `SettingsPass` collects the reads `initial_load` will make as `(name, applier)` pairs, fetches them together, then replays the appliers in declaration order. Declaring is what makes the batch exact. The same `if server_mode` / `if *CLOUD_HOSTED` / `cfg` branches that used to guard a read now guard a declaration, so the fetch asks for what this process needs and nothing else, and there is no list of setting names to keep in sync with anything. Ordering is preserved end to end: appliers run in the order they were declared, and non-setting work in the middle of the sequence keeps its place as a step, so nothing moves and nothing runs twice. Steps that need several settings at once take them together. The batch distinguishes three states where a per-setting read only ever produced two at a given call site: - a value, - genuinely unset, which several settings must see in order to restore a default when the setting is cleared, - could not be read, which must leave the in-memory value alone. Collapsing this into "unset" would let one failed query reset workspace fairness and the queue caps across a cluster. Over HTTP the reads go out together rather than sequentially, so an agent worker's settings load costs one round instead of ~36, with no new endpoint. A setting an agent may not request still resolves to unset, as the per-setting call returned for it. `reload_*` keeps working per setting for the notify path, sharing its apply half with the pass. The wrappers no caller was left using are dropped. worker startup: 50 queries -> 2 (the batch, and jwt_secret which stays its own read so the pass cannot sit between reading it absent and upserting a replacement over another replica's). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: run the pass's non-setting steps in declaration order too Review round found the settings pass had a gap: the reads were declared but the work interleaved between them still awaited inline, so it all ran before `pass.run` applied anything. `manage_audit_partitions` therefore saw `AUDIT_LOG_RETENTION_DAYS` at its compile-time default rather than the configured value, and dropped every partition past that default. An instance keeping 30 days on CE lost the 14-to-30-day band on startup and on every full-reload tick. The `STORE_AUDIT_LOGS_S3` export anchor had the same cause: the gate read `false` before the setting applied, so an env-var-enabled export never anchored and its first tick skipped the rows committed before it. `action` exists so a step keeps its place in the sequence; every remaining inline await is now one, which fixes both and leaves no phase where a read can observe a value the pass has not applied yet. Two more from the same round: A batch that fails as a whole now falls back to per-setting reads. Skipping every applier preserves known-good state on a reload tick, but a starting process has none, and would have run on compile-time defaults until the next full reload twelve hours later. `FORCE_RUBY_REPOS` is honored again: the batched url-list path parsed without the `FORCE_` check its per-setting counterpart applied, so the override was silently dropped. `load_setting_value` never had one, so the third helper was never affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: declare the object-store and worker-config steps in the pass too Two awaits were left running ahead of `pass.run`, so the settings they read were still at their compile-time defaults. The object-store reload is the one that matters: an AWS OIDC store mints its first token against an issuer built from `BASE_URL` (`oidc_ee.rs`), and with `OTEL_ENVIRONMENT` set nothing loads that before this pass does, so the store signed with the unset default, left `OBJECT_STORE_SETTINGS` empty and fell back to the ten-second retry while startup carried on. `reload_worker_config` calls `store_pull_query`, which reads the workspace fairness knobs. It happened to converge because the enabled flag re-stores the query when it changes, but it was reading defaults on the way there. Both are steps now, which is also what the earlier fix should have covered: the only await left outside a step is `pass.run` itself. Also from the same round: `fetch_settings_batch`'s doc comment had been stranded on the helper inserted above it, and the batch-failure fallback re-ran the same reads on an agent worker, where the batch already is the per-setting read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: point the setting-loader docs at functions that still exist `reload_setting` went with the other wrappers no caller was left using, but two doc links still referenced it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: decide the jwt secret in sql so the read can be batched `reload_jwt_secret_setting` generated a secret whenever its read came back absent or unparseable, and upserted it unconditionally. Two replicas booting against an empty row therefore each installed their own and rejected each other's tokens, and the same happened on a running cluster whenever the row was deleted or set to a non-string. Keeping the read next to the write kept the window narrow but never closed it, and it was the reason this one setting could not go through the settings pass. `get_or_create_jwt_secret` puts the decision in the statement instead: INSERT ... ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value WHERE jsonb_typeof(global_settings.value) <> 'string' RETURNING value First writer wins, a usable secret is never overwritten, and an empty RETURNING is how a caller learns another process's secret stands. The `WHERE` also keeps a normal startup from writing at all, which matters because `notify_global_setting_change` fires on every write to this table and an unconditional upsert would have made each start trigger a cluster-wide reload. Because the statement decides rather than the caller's read, a stale value is harmless and `jwt_secret` is now an ordinary declaration. Worker startup is a single batch round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a failed read from dropping a FORCE_ override or clearing a setting Two ways a read that did not succeed was being treated as an answer. A `FORCE_` override used to be checked before the read, so a failed read could not affect it. Moving that check into the parser put it behind a value arriving, and a failed read skips its applier, so a forced private registry fell back to the public index and a forced `settings.xml` was deleted from disk by the Maven step that follows it. Forced settings are declared as steps with no read now: the override outranks the database, so there is nothing to fetch and nothing to lose when a fetch fails. The setting loaders were passing `v.ok().flatten()` to their appliers, which turns a database error into "unset". Most appliers ignore `None`, but `apply_tag_per_workspace_workspaces` clears the workspace whitelist with it, making every workspace eligible for per-workspace tags, and `apply_fork_workspace_tag_append_fork_suffix` stores `false`. Both are also reached from the notify handlers, so a blip during a reload changed routing for the cluster. They take `?` now, as the code they replaced did by leaving the error arm empty, and the other five are converted with them so an applier that later grows a `None` branch cannot inherit the problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: route hub_api_secret through the FORCE-aware declaration `HUB_API_SECRET` lives in an `ArcSwap` rather than an `Arc<RwLock<_>>`, so it could not use `option_setting` and was declared by hand with a bare `setting` plus `parse_option_setting_value` — which is exactly the path that skips the `FORCE_` handling, so a failed read still dropped `FORCE_HUB_API_SECRET`. The rule now lives in `option_setting_with`, which takes the store closure and leaves `option_setting` a wrapper over it, so a setting held in something other than an `RwLock` reaches it too rather than having to reimplement it. The three remaining hand-written parses are `parse_setting_value`, which has no `FORCE_` handling to miss: `load_setting_value` never had the check either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-14 18:03:33 +02:00
Ruben FiszelandClaude Opus 5	6d03784d4b	fix: keep non traffic-serving processes out of coordinated restarts (#10694) ... * fix: key server_heartbeat row on hostname so restarts reuse one row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: trim announce_server_started doc to the durable constraints Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only traffic-serving processes take part in coordinated restarts Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: name every non traffic-serving mode in the restart-gate comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: narrow the restart-gate comments to claims that hold Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-14 15:10:30 +02:00
Ruben FiszelandClaude Opus 5	22eadab67d	perf: resolve the worker external IP in the background (#10697) ... * perf: resolve the worker external IP in the background `run_workers` awaited `external_ip::get_ip()` — an HTTPS GET to hub.windmill.dev — before spawning any worker, so every worker process paid that round trip before its first job pull. Measured on a CE debug build it was 120-450 ms of a ~200-500 ms startup, and behind a firewall the call does not fail fast: it burns its whole 5 s connect timeout, on every process start. That cost is per-job under EXIT_AFTER_N_JOBS. The value is informational (it is only written to `worker_ping.ip`, which the workers list displays so users can whitelist the address), so nothing needs to wait on it. It now resolves into a process-wide cache off the startup path, and `WORKER_EXTERNAL_IP` supplies it explicitly for deployments that know their egress address or have no egress at all. Until it resolves the ping carries no IP, which `insert_ping_query` now COALESCEs so a reclaimed row keeps the address the previous process wrote instead of being blanked. The main loop reports the IP as soon as it lands rather than on the next periodic tick, so a short-lived process still records it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep unknown worker IPs out of the whitelist alert Review follow-ups: - `WhitelistIp` filtered only the `'unretrievable IP'` sentinel, so the `'NO IP'` one a pending or failed lookup now leaves in the row would be offered as an address to whitelist. It filters both. - Register `WORKER_EXTERNAL_IP` in `ENV_SETTINGS` so operators can confirm from the instance settings view that it took effect. - The worker tracked whether it had reported the IP by re-reading the cache after each ping rather than remembering what the ping carried, so a lookup landing mid-ping marked it reported without it reaching the row. The value is read once and threaded through `insert_ping` / `update_worker_ping_full`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report a sentinel IP once the lookup has definitively failed Keeping the previous process's address on a reclaimed `worker_ping` row is right while the lookup is still in flight, but not once it has failed: the row would advertise an address nothing has confirmed, and the whitelist alert would offer it. A failed lookup now reports `UNKNOWN_IP`, leaving NULL to mean "in flight". `WORKER_EXTERNAL_IP` is rejected when longer than the `varchar(50)` column rather than panicking the worker on its initial ping, which is a hard failure. Adds the regression guard for the `ON CONFLICT` semantics: reverting to `ip = EXCLUDED.ip` would compile and blank every reclaimed row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the agent initial ping acceptable to older servers An agent worker routinely runs against a server of a different version, and one predating the background lookup rejects an initial ping carrying no IP — which `run_worker` turns into a panic, so a newly upgraded agent would crash-loop against it. The not-resolved-yet case goes over the wire as the sentinel instead, and the server maps it back so a reclaimed row still keeps its address while resolution is pending. Also documents `ip` as the one conditional exception to `insert_ping_query`'s "only `started_at` and `jobs_executed` survive a restart", and adds `WORKER_EXTERNAL_IP` to the README env-var table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: deliver the resolved IP to servers that only take it at registration A server predating the background lookup applies `ip` from the initial ping only, and ignores it on the periodic ones. An agent registering before its lookup resolves would therefore keep the sentinel forever on such a server, where it used to report its real address. It registers a second time once the address is known, skipping that when the address is still unknown, when the server is reached over SQL and needs no second registration, or once a job has run, since registering clears the row's current job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: re-register the resolved IP even after a job has run Gating the second registration on "this process has not run a job yet" meant an agent that pulled queued work before its lookup resolved never delivered the address to a server that only takes one at registration. No job of the worker is in flight where that runs, so the gate bought nothing beyond the last job's id, which the next job refills. Documents the two cases where WORKER_EXTERNAL_IP stops being an optimisation and becomes the only way to report an address: an agent against such a server, and a process shorter-lived than the lookup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert: drop the WORKER_EXTERNAL_IP escape hatch Supplying the address by hand skips the hub lookup, which is not something to make easy. Resolving it in the background is what keeps it off the startup path; opting out of it is a separate decision this does not need to take. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: distinguish an IP never established from one that could not be retrieved `NO IP` was doing double duty: the column default for a row whose lookup has not resolved, and the marker for one that failed. An operator reading the workers list could not tell "not resolved yet" from "this instance cannot reach the hub", and the latter is the actionable one. A failed lookup now reports `unretrievable IP`, which is also what it reported before the lookup moved off the startup path. That leaves `NO IP` meaning only "no address established", which is what an agent sends while its lookup is in flight and what the server maps back to "unresolved" — so the wire sentinel no longer collides with the failure marker, and an agent delivers the failure to a server that only reads an IP at registration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-14 14:27:21 +02:00
Ruben FiszelandClaude Opus 5	2fcce4526a	feat: add EXIT_AFTER_N_JOBS worker mode for environment cleanup (#10671) ... * feat: add EXIT_AFTER_N_JOBS worker mode for environment cleanup Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address review findings on the EXIT_AFTER_N_JOBS worker mode Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-2 review findings on EXIT_AFTER_N_JOBS Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-3 review findings on EXIT_AFTER_N_JOBS Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound WORKER_SUFFIX length and document the same-worker drain Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: validate the assembled worker name length Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-13 06:14:55 +02:00
hugocasaandClaude Opus 5	c09de594b6	feat: version resource values with history, diff and restore (#10596) ... * feat: version resource values with history, diff and restore Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: record resource versions in a trigger so direct writes are covered Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: show the selected version's value and tighten history write access Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: gate resource version recording in trigger WHEN clauses Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: clear a resource's past versions, and address review nits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: restore the displayed version and keep author attribution on pooled writes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope history to the selected workspace and gate clearing on ownership Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate restore on write access and clearing on the signed-in workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(frontend): share the version-history row between script and resource drawers Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: trim resource version history in the monitor sweep, not on write Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): match the script versions drawer shell for resource history Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(frontend): highlight version values instead of mounting monaco Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): match the script drawer's code preview presentation Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: rank version trim in one windowed pass instead of a correlated delete Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(frontend): treat the newest version as current by position Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: gate the resource version trim to an hourly sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: unnest the version row action and correct the trim cadence docs * perf: cap the history listing and use sets for reference lookup * feat: warn when a resource is written more than 60 times a minute * fix: lower the resource write advisory to 20 per minute * fix: discard stale history loads and never diff against an unread value * fix: correct the write advisory boundary and document the eviction lock * fix: read history and the live value from one snapshot * refactor: read the drawer's diff baseline from versions, not the live resource * fix: open the history drawer with no version selected * fix: disarm the clear confirmation and clear the pane when the selection moves * fix: explain the missing diff and drop a guard that can no longer fire --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 21:32:18 +02:00
Ruben FiszelandClaude Opus 5	c59b60c729	fix: keep the same_worker pin when a suspend ends without approval (#10552) ... * fix: keep the same_worker pin when a suspend ends without approval A disapproved or timed-out approval gate hands the flow back through the UpdateFlow channel with unrecoverable = true. That flag means "the previous step's worker died", and it is read by six sites. Five of them happen to want what it does here, but continue_on_same_worker and continue_with_runners do not: the worker that ran the approval step is alive, so unpinning the error handler and routing it by tag breaks the ./shared contract of a same_worker flow and can land it on a worker group that cannot run it — the same defect #10551 fixed for the three producers that hand back a live flow. Replace the boolean with StepFailureKind so the suspend producer can say "worker alive, but this failure is not the module's to handle" instead of overstating a worker death. The failed module's error policy is deliberately still bypassed: the failure is recorded against the step the gate was holding back, which never ran, so its retry would re-open the gate and its continue_on_error would skip it outright (verified: the gated step is marked Failure with a nil job id and the flow jumps past it). suspend. continue_on_disapprove_timeout remains the way to continue past a gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(flow-editor): flag that continue on error does not cover the approval gate A resolved approval is recorded against the step the gate holds back, not the step carrying the suspend, so continue_on_error never sees it: the flow still stops on a disapproval or timeout. Point users at suspend.continue_on_disapprove_timeout, which is what actually continues past a gate, whenever both settings are on and that one is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 23:59:42 +02:00
Ruben FiszelandClaude Opus 5	25084170d7	feat: make job subprocess oom_score_adj configurable (#10443) ... * feat: make job subprocess oom_score_adj configurable via JOB_OOM_SCORE_ADJ Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: warn when JOB_OOM_SCORE_ADJ leaves no gap over the worker's own score Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: drop em dash from JOB_OOM_SCORE_ADJ doc comment Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: warn on any oom_score_adj gap too small to steer the OOM killer Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 11:50:47 +00:00
Alexander Petric	e0d6dc1a19	fix: harden flow-orchestration token refresh (mint from job_perms) (#10419) ... * fix: harden flow-orchestration token refresh (mint from job_perms) * refactor: address review nits on flow token refresh	2026-07-31 00:05:15 +02:00
3 people	81b23a2ba0	feat: make the fork lineage the only deploy relationship (#10410) ... * feat: make the fork lineage the only deploy relationship `workspace_settings.deploy_to` (2023) and `workspace.parent_workspace_id` (2025) both expressed "which workspace does this one deploy into". Fork creation and dev-workspace attach seeded both, but nothing kept them in agreement, so every reader picked one and they disagreed. Drop `deploy_to`. A migration folds surviving pairs into the lineage: a sole claimant on a target with no dev workspace becomes that target's dev workspace and keeps its own job tags, while many-to-one pairs become plain forks. Pairs that the lineage cannot express -- dangling target, self-reference, chain, mutual -- are reported and left unlinked. Job tags were never lineage-aware: `per_workspace_tag` mapped any parented workspace to its parent while `$workspace` interpolated the raw id, so a fork running a script tagged `<tag>-$workspace` produced a tag no worker serves and the job queued forever. Both paths now resolve to the nearest ancestor whose id an admin would provision workers for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve unconvertible deploy links and sweep tag caches on reparent Review findings on the deploy_to unification: - convert chains instead of discarding them, and keep whatever the lineage cannot express in workspace_deploy_to_unmigrated so the down migration can restore it - ignore soft-deleted workspaces when choosing between a dev workspace and a plain fork; an archived claimant was demoting live pairs - mirror attach_dev_workspace's git-sync strip, which the migration skipped - sweep the tag cache over whole subtrees on rename and delete: tag resolution now walks ancestors, so a nested fork kept a tag nothing serves - call a dev workspace a dev workspace in the settings copy - redirect a root away from ?tab=deploy_to instead of rendering an empty target Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: detect lineage cycles and record archived links in the deploy_to migration Second review round on the unification: - detect cycles over the lineage as it would exist after conversion, not over the deploy_to graph alone: a root whose target was one of its own forks closed a loop that no deploy_to edge revealed - record an archived source's link instead of filtering it out entirely, which dropped it with the column - treat a fork whose deploy_to merely repeats its parent as redundant rather than reporting every pre-existing fork as unmigrated - read the row count from the lineage update rather than the git-sync one - sweep the tag cache when archiving a dev workspace, the last site that mutates is_dev_workspace without one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve $workspace on preprocessed flow tags regardless of $args Third review round on the unification: - a flow tag containing only `$workspace` skipped interpolation entirely on the preprocessed path, because the branch that ran it keys on `$args`. The raw tag was written back and named a queue no worker serves. Resolve `$workspace` before the branch and leave `$args` to it. - record the new table's foreign key in the schema summary - describe what the archive tag sweep actually does: the dev flag is cleared for any archived workspace, which is why it is unconditional Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the deploy_to leftovers table only when it holds something * fix: sweep tag caches on archive only where the dev flag actually changes * feat: broadcast lineage changes and walk ws_specific ancestors only - propagate tag-cache invalidation across processes over notify_events: the cache is per-process, so replicas kept resolving stale lineage for the TTL. The listener clears the whole cache rather than tracking ids, since a single mutation invalidates an unbounded set of descendants and lineage changes are rare admin actions. - narrow list_ws_specific_versions to ancestors: walking down as well made a root fan out over its entire live fork subtree, and each member costs an identity lookup plus an RLS switch and probe. Ancestors are bounded by the fork depth limit. - probe the leftovers table unqualified so rollback restores on a PG_SCHEMA install, where search_path is not public - drop the nativets client method for the removed edit_deploy_to endpoint Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let a prod see its dev workspace in ws_specific, and stop the walk oscillating Descending into plain forks made a root fan out over its whole live fork subtree, but a dev workspace is the paired editable environment rather than a throwaway copy, so a prod should still see it. There is at most one per parent and attach rejects nested dev chains, so that edge stays bounded. The edges run both ways, so the recursion never converged: it bounced parent<->dev until the depth cap on every call, 33 rows for a two-member set. A visited-path guard ends the walk when nothing new is reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep dev pairings unnested, gate the delete broadcast, cover the ws_specific walk Fifth review round: - a root that already owns a dev workspace no longer converts: linking it under its deploy target would leave that dev nested beneath a fork, the shape attach_dev_workspace refuses to create. The link is preserved instead. - broadcast a lineage change on delete only when descendants are orphaned. Deleting a leaf, which ephemeral fork churn does constantly, changes nobody else's resolution and was making every replica drop its whole tag cache. - call list_ws_specific_versions in a test. plpgsql defers everything past a raw parse to the first call, so replaying the migration only proved it parses. - use unwrap_or_default for the descendant sweeps, which run after the transaction has committed; a transient failure must not fail the request - trim the traversal comment to the four-line limit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cache the renamed tally query and clear instance alerts on conversion The integration test's query was never cached: `cargo sqlx prepare` without --all-targets skips test targets entirely, and renaming its fixture workspace changed the query text. Regenerated with --all-targets --features all_sqlx_features,private, which is what lets the EE-gated otel test compile. Also from review: - clear error_handler_fallback_to_instance_alerts on converted workspaces. Dispatch ignores it once a parent exists, but the settings page keeps submitting the stored true, which the API rejects on a fork. - restore the schema summary row to the file's name: columns format and put it back in alphabetical order Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never cache an unresolvable tag workspace, and unadvertise the removed endpoint - lookup_tag_workspace cached a "no row" result as self-resolution. A rename resolves the new id before its row lands, so a fork could be pinned to its own wm-fork-* id -- which nothing serves -- for the whole TTL, and its schedules kept re-pushing onto that dead tag. Fall back for the call without caching, matching how the error path already behaved. - change_workspace_id swept its children but never itself. Sweep the new and old ids and broadcast unconditionally, since a rename always changes lineage. - openapi-deref.{json,yaml} are served to clients via include_str!, so they were advertising edit_deploy_to after it started 404ing. The audit-action enum keeps the entry: historical rows still carry it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: align the served YAML spec with the JSON one and correct two comments - the YAML deref lost the removed path but kept deploy_to on get_settings, so the two served specs disagreed. Both are now identical. - the rename-sweep comment blamed cached-unresolvable lookups, which the same commit stopped caching. The real reason is that workspace ids are reclaimable, so a new id can carry a previous occupant's resolution. - the instance-alert comment claimed the settings page submits the stored true and gets a 400. It hides the option on a fork and sends false; the hazard is the value outliving the pairing and re-enabling alerts after a detach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 82da6cb2bafeda18acd6b70c599013a12117ecb0 This commit updates the EE repository reference after PR #694 was merged in windmill-ee-private. Previous ee-repo-ref: f9ddf6a75aa13d1c13a3d7216a361a96f75ca435 New ee-repo-ref: 82da6cb2bafeda18acd6b70c599013a12117ecb0 Automated by sync-ee-ref workflow. * fix: grant the deploy_to preservation table to the windmill roles * test: drop the one-shot migration tests, keep the ws_specific execution guard The two conversion tests replayed the migration against the fully-migrated schema, which is not how it runs -- in production it runs mid-sequence against the schema as of that point. A later migration touching workspace or workspace_settings would break them without breaking anything real, and sqlx checksums already freeze a released migration. They earned their keep finding the archived-claimant and nested-dev cases during development; there is nothing left for them to guard. list_ws_specific_versions is different: it is live, no caller exercises it, and plpgsql only parses a function body until first call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: invalidate a reclaimed fork id cluster-wide without flushing every entry Gating the delete broadcast on orphaned descendants stopped leaf churn flushing every replica, but fork ids are reclaimable: the deleting process invalidated locally while every other replica kept the old parent for the TTL, so a job pushed in a recreated fork routed to the previous parent's tag. The broadcast payload now carries meaning. A workspace id drops that one entry, used for leaf deletion where exactly one id changed what it denotes. The `*` sentinel drops everything, used for attach, detach, archive, rename and deletions that orphan descendants -- reshaping a subtree no single id names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name the right broadcast for each invalidation case * docs: attach does invalidate the tag cache; the resolver walks the whole chain --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-30 14:20:27 +00:00
Ruben FiszelandClaude Opus 5	4d3ff0299f	feat: mark failed jobs as resolved so handled failures stop showing red (#10319) ... * feat: mark failed jobs as resolved so handled failures stop showing red Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: constrain auto-resolve to the proven retry chain and honor resolved filter everywhere Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply resolved filter to queue-union, concurrency and delete paths, bound note Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: sweep resolutions on workspace delete, verify helper args, enforce UI limits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count resolution note in characters on both sides of the API Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: skip the queue lookup for cancel-all under the resolved-only filter Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: converge retry auto-resolution from either commit order, keep notes on re-resolve Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the idempotency claim on the retry auto-resolve sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: gate resolution notes and attribution behind enterprise, add note popover Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hide resolution from operators, exclude flow steps, enforce EE licence at runtime Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: add job_resolution.automatic to the summarized schema Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve stored attribution when re-resolving without a valid licence Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: condense the attribution-preservation comment to four lines Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: validate resolution notes by code point instead of a UTF-16 maxlength Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the resolution popover open when a note is rejected Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: offer to resolve the original failure after a successful re-run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: verify supersession server-side and stop re-runs overwriting notes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply tag scope to the superseding run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: exclude obscured cross-workspace runs from resolution actions Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 09:36:27 +02:00
3 people	30d8104edc	feat: alert on expired online license key (#10295) ... * [ee] feat: alert on expired online license key Wire alert_on_online_license_expired into the periodic monitor loop (verify_license_key_f), server-mode gated so only servers report to the alerts table and critical channels. Add the ee_oss stub so the enterprise-without-private build still compiles. Closes the gap where an expired online (renewable) license force-set externally (env var / CI / k8s) halted all jobs with only a stdout tracing::error! and no critical alert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: point ee-repo-ref at license-expired-alert EE branch Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 9e63619688588fda434dfa06ed585f050e75ee67 This commit updates the EE repository reference after PR #682 was merged in windmill-ee-private. Previous ee-repo-ref: 7847711567a9c66da6a5c85d176b8cacc5aa7a7f New ee-repo-ref: 9e63619688588fda434dfa06ed585f050e75ee67 Automated by sync-ee-ref workflow. * chore: bump ee-repo-ref to license-expiry alert review fixes Point at the EE follow-up (windmill-ee-private#683): cross-replica dedup via acquire_lock and no false recovery on malformed key replacement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eb8733a36010a7509b438903e95edfa293da079b This commit updates the EE repository reference after PR #683 was merged in windmill-ee-private. Previous ee-repo-ref: 562a306d9b66624b28ff90958f8b45d8db5481b9 New ee-repo-ref: eb8733a36010a7509b438903e95edfa293da079b Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-24 10:53:24 +02:00
Ruben FiszelandClaude Opus 4.8	f02df7fc45	feat(monitor): make between-steps zombie flows hand-recoverable (#10287) ... * feat(monitor): make between-steps zombie flows hand-recoverable When a worker is OOM-killed mid state-transition, the flow is reaped as a between-steps zombie (children all success, module still InProgress). We do not auto-recover (a re-driven transition can OOM again), so instead: - Append actionable recovery guidance to the cancellation reason when the reaped step's state is derivable (every child a success completion): which step, iterations completed, raise memory then restart-from-step (UI + API). - Restart-from-step now reuses a zombie step verbatim (InProgress with all children successful) and restarts from the next step, so no completed child re-runs; downstream steps re-derive its result from flow_jobs on demand. - Cast flow_status ::text in the reaper query: reading the jsonb column as Box<str> included the binary version byte and silently failed FlowStatus parsing (disabling the restart-not-yet-started branch since the v2 migration). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): only reuse a between-steps zombie step that provably finished Address review findings on the zombie-restart reuse path: - Require structural completeness (FlowStatusModule::is_between_steps_complete): a serial for-loop / branch-all reaped mid-fan-out has an all-success prefix but unrun remaining iterations, so the cursor must sit on the last element; while-loops are never derivable (continuation is a post-iteration condition). Parallel containers preallocate all children, so success alone is conclusive. Shared by the monitor guidance and the restart resolution. - Decline reuse when the step carries stop_after_if / stop_after_all_iters_if: those predicates decide whether downstream steps run, and reuse would bypass them; such a step re-runs instead. - Decline reuse when the zombie step is the last module (advancing past it lands on the failure step); it falls back to the existing re-run path. - Unit tests for is_between_steps_complete and an integration test asserting a mid-iteration serial-loop zombie is re-run, not reused. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): align zombie recovery guidance with restart eligibility Address CI review findings: - Exclude skip_if / suspend / sleep (not just stop predicates) from reuse via FlowModule::allows_zombie_reuse, so a skipped/suspend-armed step is never synthesized as Success (which would strand a restart waiting on an approval it never armed). - The reaper does not load the flow definition, so it cannot know whether restart will reuse or re-run a given step; reword the guidance to state both outcomes (reuse where derivable, re-run for the flow's last step or one carrying a stop/skip condition, approval, or sleep) instead of promising "no re-run". - Make the mid-iteration regression test exercise the cursor-completeness guard: a downstream step makes the loop non-final, so reuse is prevented only by the guard; a truncated loop result would then fail the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): never let zombie reuse swallow a nested restart request A nested restart (RestartedFrom.nested) descends into the restart step's child to re-run an inner step. For an eligible zombie BranchOne/Subflow the outer branch_or_iteration_n is None, so reuse fired, skipped the container, and the explicitly requested inner step never re-ran. Thread the presence of a nested chain into restarted_flows_resolution and decline reuse when set. Regression test added (RED without the guard: the nested target is reused instead of re-run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): don't auto-requeue preprocessor zombies as unstarted flows The ::text parse fix re-activated the "hasn't started yet, restart it" branch, but its `modules[0] == WaitingForPriorSteps` check also matches a flow whose preprocessor is still InProgress (step == -1, first module waiting). Requeuing such a flow re-runs the preprocessor, duplicating side effects / repeating the OOM. Gate the branch on FlowStatus::is_not_yet_started, which also requires the preprocessor (if any) to be WaitingForPriorSteps. Unit-tested. Also drop the numbered procedural narration from the happy-path test comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): only emit restart guidance for restartable (deployed, top-level) flows The recovery guidance points operators at the run page's "Re-start from" button and the restart API, but both require a top-level deployed flow: a preview has no flow path (the button is hidden, the API 400s) and a subflow child restarts via its root, not itself. Gate the guidance on runnable_path IS NOT NULL AND parent_job IS NULL so previews/subflows keep the existing wording instead of being told to use a button/endpoint that isn't there. Verified end-to-end: a reaped preview gets no RECOVERY block, a reaped deployed flow does. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): gate recovery guidance on kind='flow' to match the restart surface Addresses review nit: a pathful editor preview (kind='flowpreview' with a runnable_path) satisfied the previous runnable_path check but the run page only renders the "Re-start from" button for kind='flow'. Match that condition exactly so previews/singlestepflow keep the plain wording. Verified end-to-end: a reaped pathful preview now gets no RECOVERY block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): disable zombie reuse for raw-flow (editor preview) restarts A JobPayload::RawFlow restart queues the request's current, possibly EDITED, definition, but restarted_flows_resolution validates reuse against the completed job's STORED definition. For an eligible preview zombie, editing the restart step and restarting from it would synthesize Success from the old children and skip the edit. Thread allow_zombie_reuse into the resolver (true only for JobPayload::RestartedFlow, which queues the stored definition) and decline reuse for raw-flow restarts. Regression test added (RED without the guard: the edited step is skipped and the old result is reused). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(sqlx): add offline cache for zombie_flow_recovery test queries The integration test's UPDATE v2_job_completed queries had no .sqlx entry, so the CI SQLX_OFFLINE build of the test failed to compile. Regenerated with --all-targets --features deno_core,quickjs to capture the test-target queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(monitor): drop procedural narration from the raw-flow zombie test Per AGENTS.md (comments record constraints, not narration): remove the two step-describing comments the reviewer flagged; the test doc comment already carries the durable rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): restrict zombie reuse to monitor-reaped flows The reuse predicate matched the InProgress/all-children-success shape without checking provenance, so an ordinary force-cancel at the same boundary (a child succeeded before its parent transition landed) would also be reused, dropping the usual restart-from-step re-run. Gate reuse on canceled_by = 'monitor' (the username the zombie reaper cancels with). Regression test added (RED without the guard: a user-cancelled flow reuses the child instead of re-running it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): reuse zombie step on Some(0) too, so the run-page button works The run page's "Re-start from" button always sends branch_or_iteration_n = 0 (never omits it), but reuse only fired for None, so the exact UI path the recovery message points to would re-run the children instead of reusing them. Treat a whole-step restart (None or Some(0)) as reuse-eligible; Some(n>=1) keeps the explicit partial-container restart. Verified against the live EE restart API with branch_or_iteration_n=0: all loop-iteration child UUIDs are reused. Happy- path test now sends Some(0) to match the button. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 19:09:28 +02:00
Ruben FiszelandClaude Opus 4.8	fa3644281f	fix(monitor): diagnose zombie-flow OOM on the transition worker, not q.worker (#10286) ... handle_zombie_flows() joined worker_ping on the flow's queue-row worker (q.worker) only. In nested/subflow/forloop cases that worker is frequently NOT the one that performed the flow's final state transition, so all OOM detection ran against a healthy bystander and the diagnostic wrongly told on-call to chase a deadlock / SIGQUIT. Change 1: when q.worker itself looks healthy, additionally search worker_ping for a plausible culprit — a different worker on the same pod (worker_instance) or worker group whose last ping clusters around the flow's last_ping and has since gone silent (the signature of a worker OOM-killed mid state-transition), picking the one closest in time to the transition. If found, the message names that worker and routes to the OOM explanation. The extra query runs only on the rare zombie path and fails soft (warn + fall back) so it never blocks the cancel. Change 2: reorder the healthy-q.worker hint to lead with "check whether a different worker on the same pod/group was OOM-killed around {last_ping}", keeping deadlock/SIGQUIT as the secondary possibility. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 11:40:23 +00:00
Ruben Fiszel	ddec2abbb3	feat(jobs): cap total queued jobs per workspace on cloud (#10218) ... * feat(jobs): cap total queued jobs per workspace on cloud A workspace could flood the queue with an unbounded number of jobs across many concurrency keys and scripts (or keyless jobs), which the per-key cap from #10197 does not bound. Add a companion instance-wide ceiling on a workspace's total queued jobs. check_workspace_queue_cap rejects a push once the workspace has WORKSPACE_MAX_QUEUED_JOBS (default 20000, superadmin-configurable, 0 to disable) jobs queued, cloud-only and runtime-gated on CLOUD_HOSTED like the per-key cap. It runs on every push, so it applies even to premium workspaces and catches parallel for-loop floods. Jobs already queued still drain; only new pushes past the ceiling are rejected, so an in-flight flow only fails to push further work while at the ceiling. The setting loader self-gates on CLOUD_HOSTED so it is never loaded off cloud, from initial load or a settings-change reload. The depth count is bounded by the cap via LIMIT so a runaway backlog never costs an unbounded scan on the push path. * docs(jobs): note the workspace cap is a soft ceiling and the depth helper is count-only Records the two review points as constraints: the cap does not serialize admission (a soft ceiling by design, like the per-key cap), and workspace_queue_depth is pub only for the test, returns a count not job data, and leaves authorization to the caller.	2026-07-20 22:09:59 +02:00
4 people	11fda89b52	feat(telemetry): generic feature-usage telemetry with AI session metrics (#10200) ... * feat(telemetry): add generic feature_usage table and batched logging endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(telemetry): log AI session usage events and document them in telemetry settings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): use escape sequence instead of literal NUL bytes in buffer key Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): validate dimensions, decouple retention, keepalive flush Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): allowlist feature-usage dimensions and index retention scans Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): pin tool-name allowlist and deploy session attribution Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(telemetry): route AI chat usage through feature_usage and drop ai_chat_usage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(telemetry): slim dimension validation to registered kinds plus key shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): backfill ai_chat_usage into feature_usage before dropping it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): disclose provider and model identifiers in telemetry settings text Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): issue all flush chunks before awaiting so pagehide keeps them Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6306c072a50937ea9af44a5bcf42345543207486 This commit updates the EE repository reference after PR #672 was merged in windmill-ee-private. Previous ee-repo-ref: 964f242a0eb44db7f7d26636cc8d76aeabea2b73 New ee-repo-ref: 6306c072a50937ea9af44a5bcf42345543207486 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-07-20 20:56:57 +02:00
3 people	87be041c09	fix(git-sync): avoid percent-encoded colon in git-sync hub script path (#10213) ... * fix(git-sync): avoid percent-encoded colon in git-sync hub script path The git-sync init/detection hub script slug contained a colon stored as `%3A` in the run-by-path URL. The generated API client re-encodes path params with encodeURI, turning `%3A` into `%253A` (double-encoding). Some hardened reverse proxies / WAFs reject double URL-encoding and return a bare 400 before the request reaches Windmill, breaking git-sync repository detection on those instances. The hub resolves scripts by numeric id and ignores the slug, so dropping the colon from the slug is behavior-neutral (same script, same id-keyed worker cache) while producing a colon-free run URL. Also force-cache GIT_SYNC_PULL_SCRIPT_PATH at build alongside LATEST_GIT_SYNC_SCRIPT_PATH so the backend-driven pull script is always baked into the image for airgapped workers, instead of relying on an incidental hubPaths.json overlap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for git-init slug match fix Pulls in windmill-ee-private#676 so the EE is_git_init_script check matches the colon-free git-init hub slug (GitHub App token grant for git-sync jobs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for git-init slug helper + test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to a3adea1ffb406e709cc480871df58fab6c51aca1 This commit updates the EE repository reference after PR #676 was merged in windmill-ee-private. Previous ee-repo-ref: cef4e008ef62dec434aa9bb3ec783db8aff6a1c1 New ee-repo-ref: a3adea1ffb406e709cc480871df58fab6c51aca1 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-20 20:36:34 +02:00
Ruben Fiszel	71f2d47cb4	feat: cap queued jobs per concurrency key on cloud (#10197) ... * feat: cap queued jobs per concurrency key on cloud * fix: close preprocessed-flow bypass and bound concurrency cap scan * fix: only cap concurrency keys with an active concurrent_limit * chore: only load concurrency key cap setting when cloud hosted * fix: reject queued-job import on cloud	2026-07-20 12:33:40 +02:00
Ruben FiszelandClaude Opus 4.8	f6e36f862e	chore(schedules): lower reconciler re-arm back-off cap to 8 passes (#10182) ... Follow-up to #10179. The exponential back-off between reconciler re-arm retries of a persistently-failing schedule capped at 32 passes (~2.7h at the default 5-min reconcile cadence). Lower the cap to 8 (~40min) so a schedule fixed out of band (a lapsed license renewed, a bad cron corrected directly in the DB) auto-recovers within a few passes, while still cutting the retry rate sharply versus retrying every pass. Fixes via the UI/API re-arm immediately and are unaffected. Fixes WIN-2198 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 08:49:41 +02:00
Ruben FiszelandClaude Opus 4.8	c82056cfde	fix(schedules): stop disabling schedules on transient push errors (#10179) ... * fix(schedules): stop disabling schedules on transient push errors A scheduled flow whose next-occurrence push failed after retry exhaustion used to be disabled, killing a healthy schedule over a transient DB blip (pool contention, statement timeout). Now that the unarmed-schedule reconciler exists (#10174), transient failures no longer disable: the current occurrence runs to completion and the reconciler re-arms the next occurrence once this run leaves the queue. In the flow schedule-push path after retry exhaustion we now branch on the error: QuotaExceeded/NotFound still disable (the schedule's own fault, and rearm_schedule would otherwise leave them enabled-yet-unarmed forever), while transient errors are only reported and the flow continues. The previous iteration returned a SchedulePushZombieError to force a zombie restart; that is removed, because zombie detection cancels (does not restart) same-worker flows, so it would have lost the current run of a same-worker scheduled flow. The now-obsolete SchedulePushZombieError type and its catch in worker.rs are deleted. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(schedules): back off and surface repeated reconciler re-arm failures The unarmed-schedule reconciler retried a schedule that could not be re-armed on every pass, forever, logging only to the server. With the flow schedule-push path no longer disabling on non-transient errors, a persistently-broken push (bad stored cron/timezone/args, lapsed license key) now stays enabled and would spin in that loop silently. The reconciler now tracks consecutive re-arm failures per schedule: exponential back-off (2, 4, 8, … passes, capped) between retries so a broken schedule is not hammered, and after 3 consecutive failures it surfaces the cause once (records schedule.error + raises a critical alert) without disabling. Both reset the moment the schedule re-arms, which also clears the recorded error. Verified end-to-end on a running server: a flow schedule with a corrupted cron stays enabled, retries back off, the error is surfaced after the third failure, and it re-arms and clears the error once the cron is fixed. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 22:48:15 +02:00
3 people	9762089fcb	fix(schedules): re-arm enabled schedules left with no queued occurrence (#10174) ... * fix(schedules): re-arm enabled schedules left with no queued occurrence * fix(schedules): lock schedule row while re-arming and report outcome * fix(schedules): make reconcile lock cancellation-safe, re-check armed under lock Address review feedback on the schedule reconciler: - Use a transaction-scoped advisory lock (pg_try_advisory_xact_lock) instead of a session-scoped one. monitor_db runs under a 600s timeout; on cancellation a session lock on a pooled connection would be stranded, wedging reconciliation on every replica. An xact lock releases when its transaction is dropped. - rearm_schedule re-checks for a queued occurrence under the row lock and returns NoOp if already armed, closing the scan→lock window that could double-push across a cron boundary. Add a regression test. - Make reconcile_unarmed_schedules private (its only caller is in monitor.rs) and document its system-only contract. - Log the disable only after the guarded UPDATE actually disables the schedule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(schedules): never disable from reconciliation and cap re-arms per pass --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-17 18:08:10 +02:00
3 people	396fb1c475	feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs (#10139) ... * [ee] feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs Add `insecure_upstream_hosts` and `upstream_ca_certs` to the HTTP Request Tracing settings so the OTEL tracing proxy can reach internal endpoints with untrusted or private-CA certificates while keeping them traced. Wires the two settings through the worker config and live reload, adds the inputs to the instance settings UI, and pulls in the rustls upstream-client deps (hyper-rustls/tokio-rustls/rustls/ rustls-native-certs/rustls-pemfile; hyper-http-proxy switched to its rustls feature). The proxy-side implementation lives in the companion EE PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref to otel_ca companion commit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(otel-tracing-proxy): expose new fields in declarative config; bump ee-ref Addresses code-review findings: - Add `insecure_upstream_hosts` and `upstream_ca_certs` to the declarative `OtelTracingProxySettings` in instance_config.rs so operator/GitOps-managed installs can set them and reconciliation no longer drops values saved via the UI. - Restore the trailing newline on ee-repo-ref.txt and bump it to the companion EE commit carrying the strict host-matching / port-ordering fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 This commit updates the EE repository reference after PR #665 was merged in windmill-ee-private. Previous ee-repo-ref: 49f458e4446395e98915c220baa757ab3b2ed2d8 New ee-repo-ref: 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-17 15:57:00 +02:00
3 people	51d8db6602	feat: automatic git-to-windmill sync (polling, webhooks, in-app PRs + checks) (#9552) ... * docs: add design doc for automatic git-to-windmill pull sync * docs: add migration plan and implementation phases to git-sync pull design * feat(git-sync): add auto_pull settings schema and pull enqueue primitive Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings (workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant, and should_pull/effective_poll_interval_s helpers with unit tests. Exports the EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(git-sync): poll repos and auto-pull new commits into the workspace Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed, single-replica via advisory lock) git ls-remotes each auto-pull-enabled repository ~every minute and enqueues a pull when the tracked branch moves, reusing the {workspace_id}:git_sync concurrency key so pulls serialize with in-flight push commits. - windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App repos are skipped here and will sync via webhooks (phase 2). - monitor.rs: poll/reconcile/persist with optimistic sha advance and failure status; targeted jsonb update so concurrent settings edits aren't clobbered. - edit_git_sync_repository: preserve server-owned auto_pull state on UI save. - openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field. - frontend: per-repo "Automatically deploy changes from Git" toggle with last sync status; demote the GitHub Actions link to an advanced CI option. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic OSS side of phase 2 auto-pull webhooks: - edit_git_sync_repository creates/removes the repo webhook on save (EE-gated, best-effort → falls back to polling). - monitor poller now delegates to the shared windmill_git_sync reconcile/persist helpers (also used by the webhook receiver), removing duplicated logic. - export the shared reconcile/persist/failure helpers; bump EE ref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump EE ref for phase 3 in-app PR creation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(git-sync): show webhook vs polling status on the auto-pull toggle When a repo has an active webhook (auto_pull.webhook_id set), the status line reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(git-sync): post PR diff check on dry-run completion (phase 4) Worker completion hook in process_completed_job: when a DeploymentCallback job carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse and patch the GitHub check run with the diff summary (success/neutral/failure). Export enqueue_git_pull_dry_run; bump EE ref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump EE ref (drop unused GHES webhook_secret) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(git-sync): defer phase 4 PR diff checks (OSS side) Remove the worker completion hook that posted the PR check run, drop the enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref. Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)" This reverts commit 0137d3ca48. * chore(git-sync): point EE ref at restored phase 4 commit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump EE ref for clone_ref dry-run Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump init-repository hub script to v28784 Picks up the clone_ref param (windmill-integrations#158) so the phase 4 PR-check dry-run can clone the PR head. Backward compatible; manual pull/push and the automated pull/poller/webhook all move to the same published version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump EE ref for auto-pull admin-permissioning fix Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump EE ref for superadmin pull fallback Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): refresh auto-pull tooltip; bump EE ref for webhook secret encryption The auto-pull toggle tooltip claimed GitHub App repos would sync via webhooks "in a future update"; webhook delivery now works, so describe the webhook-vs-polling behavior accurately. Bump the EE ref to pick up encrypting the webhook HMAC secret at rest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(git-sync): poll app-backed repos in auto/polling mode The auto-pull poller skipped app-backed repos (the ls-remote head check can't authenticate a tokenless URL), so auto- and polling-mode app repos never synced when their webhook wasn't live. Wire the poller to fetch the head via the GitHub API for app repos and reconcile. Bump the EE ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(git-sync): auto-pull UI — direction split, delivery mode, fallback notice Reorganize the repository card into two clearly labeled directions: "Push to Git on deploy (Windmill → Git)" and "Pull from Git (Git → Windmill)". In the pull section: - new connections default to auto-pull enabled (webhook with polling fallback); existing repos load with auto-pull off and are unchanged - a Delivery selector chooses "Webhook with polling fallback" or "Polling only (air-gapped)" - a notice surfaces webhook_error when delivery falls back to polling - a reminder to remove any pre-existing GitHub Action that pushed into Windmill, to avoid conflicting double-syncs Adds the webhook_error field to AutoPullSettings (+ openapi) and bumps the EE ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(git-sync): clearer push indicator + gate webhook delivery to app repos - Push-on-deploy is shown with a check icon + concise line (via the shared GitSyncModeDisplay, restyled from the oversized "Sync:" text); the setup wizard reuses it without the check (pre-save preview). - The delivery-mode selector only shows for GitHub App-backed repos; token-based repos show a "webhooks require the GitHub App (managed or GHES)" note with a docs link and poll instead. Bumps the EE ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(git-sync): fork auto-sync (phase 5) + live deploy check (phase 6) Phase 5 — fork auto-sync configured at the parent (replaces the *-to-forks GitHub Actions): - Add fork_open_prs + fork_pull_sync to GitRepositorySettings (openapi + UI). - UI: two "Forks of this workspace" toggles in the repo card, gated on app-backed and not-a-fork; serialize the flags on save. - On fork creation, strip the inherited auto_pull block (and fork_* flags) from the copied git_sync repo: a fork must not carry the parent's webhook id (it would delete the parent's hook on disable) or self-poll on top of the parent's fan-out. Push-direction config + installation are still inherited unchanged. Phase 6 — live deploy status check on the commit (Cloudflare-style): an in-progress "Windmill" check on the head commit that flips to "Deployed N changes"; completion handled by the generalized git-sync check hook. Bump EE ref for the phase 5-6 EE implementation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * chore(git-sync): bump EE ref for PAT auto-pull mode normalization Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): address PR review findings - webhook_secret: redact from the settings API response and Debug output (still persisted encrypted); it's a server-only HMAC key the UI never needs. - poller: honor each repo's effective poll interval (relaxed ~10 min when a webhook is live) instead of probing every ~60s tick. - settings save: roll back a just-created webhook if the settings transaction doesn't commit, so a failed save can't orphan a hook. - auto-pull head check: fail SSH remotes with an actionable message (background polling has no SSH identity) instead of a confusing ls-remote error. - deploy/PR check summary: a pull result carrying neither changes nor a settings diff now falls back to the unsummarized path instead of a false "in sync". - UI: reset isGithubApp on resource change / failed fetch so webhook + fork controls can't show for the wrong repo. - tests: cover parse_git_sync_changes and format_change_list edge cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): correct feature gating for OSS builds - monitor.rs: keep the AUTO_PULL_LAST_POLL static, slack const, and poll_git_auto_pull_inner all behind #[cfg(feature = "private")] (an inserted static had split the cfg off the function, ungating it in OSS builds). - edit_git_sync_repository: the webhook create/rollback block references windmill_common::git_sync_ee (private module), so gate it on all(enterprise, private) instead of enterprise only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * chore(sqlx): cache workspace_diff query pulled in from origin/main Re-merged origin/main (advanced past the earlier merge); regenerate the offline sqlx entry for the new workspace_comparison test query so SQLX_OFFLINE builds (cargo_test) pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): address Codex review findings (webhook cleanup on delete) - Deleting a git-sync repository now tears down its managed GitHub webhook (deletion bypassed the sync_repo_webhook lifecycle, orphaning the hook so GitHub kept delivering to the instance). - Worker completion hook rolls back the optimistic auto-pull sha on job failure (OSS side of the EE change) + caches the new marker query. Bump EE ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): delete repo webhook after the removal commits Codex re-review nits: - delete_git_sync_repository deleted the webhook before the settings transaction committed; a failed save would then leave the repo pointing at a hook that no longer exists (sync_repo_webhook treats a set webhook_id as live and won't recreate it). Capture the hook id, commit the DB removal, then delete the hook. - Reword a fork-copy comment to drop drafting-history wording per AGENTS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): reconcile the edit-path webhook after the settings commit Codex nit: edit_git_sync_repository ran sync_repo_webhook before the transaction committed. The rollback only covered created hooks, but sync_repo_webhook also deletes a hook on disable/switch-to-polling — a commit failure then left the DB with a webhook_id whose hook was already gone (and it wouldn't be recreated). Save + commit first, then reconcile the webhook against the durable config and persist any hook id/secret change (best-effort). Bump EE ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): preserve webhook secret on whole-config save + default on visible add Codex nits: - edit_git_sync_config saved the client config verbatim, so the webhook_secret redacted from the GET response would be dropped (breaking delivery). Preserve server-owned auto-pull state (webhook id/secret, synced sha, last status) per repo from the existing settings, matching edit_git_sync_repository. - addSyncRepository (the visible add path) didn't set the auto_pull default, so new sync repos added from the UI came up with auto-deploy off. Match addRepository's default (webhook + polling fallback). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * refactor(git-sync): drop fork_pull_sync (parent-level keep-forks-in-sync) Removes the "Keep forks in sync with the tracked branch" toggle and its fan-out. Pulling the tracked branch straight into every fork was the inconsistent piece; the consistent model is per-fork branch sync (each fork tracks its own wm-fork/** branch), which is a separate follow-up. fork_open_prs is kept. Also tightens the fork toggle-section spacing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): detect dev workspaces in CLI fork branch derivation isForkWorkspace / computeGitSyncDeployBranch keyed off the wm-fork- id prefix. Dev workspaces are forks with a custom, prefix-less id, so their wm-fork/** branch was never derived or created. Detect them via parent_workspace_id too (which the backend already passes), mirroring the backend's `parent.is_some() || wm-fork- prefix` rule. Pairs with the hub-script clone-flag fix (windmill-integrations#163); both take effect once the CLI is released and the pinned version is bumped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): reconcile webhooks on full-config save edit_git_sync_config preserved server-owned webhook fields but never created or deleted the managed GitHub webhook, so enabling auto-pull through the whole-config endpoint only polled, and disabling or removing a repo left an orphan hook still delivering. Mirror the per-repository endpoint: after the commit is durable, reconcile every saved repo's webhook (sync_repo_webhook) and delete the hooks of repos the save removed, including the clear-whole-config case. Addresses the Codex nit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): address Codex nits (webhook orphan on cleared auto_pull, fork detection) - edit_git_sync_config: also delete a repo's old webhook when the save drops the repo OR clears its auto_pull. Webhook fields are only preserved onto a Some auto_pull, so a save that present-but-clears a repo would otherwise orphan its hook. - GitSyncRepositoryCard: isFork now uses parent_workspace_id OR the wm-fork- prefix (was AND), matching the backend/CLI rule, so prefix-less dev workspaces are detected as forks and don't show the parent fork-PR toggle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * docs(git-sync): update design doc for the dropped fork_pull_sync Phase 5 documented "Keep forks in sync with the tracked branch" (fork_pull_sync) and its fan-out as implemented; that feature was removed. Rewrite the section to reflect what ships (fork_open_prs), note the drop + the per-fork-branch follow-up, and remove the stale fan-out mentions elsewhere. Addresses the Codex nit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): redact webhook secrets from workspace export; fix doc endpoints - Export (P1): strip the server-owned auto_pull state (webhook secret/id/error + synced sha + last pull status) from git_sync before it is written into an export's settings.json for both settings formats. The HMAC webhook secret must never leave the server (matching the GET-settings redaction), and a re-imported workspace must not inherit another install's hook/sync state. - Docs: the webhook receiver is a single per-workspace endpoint /api/w/{workspace}/github_app/webhook (host-aware for managed + self-managed); update the stale push_webhook/{id} and instance-global /api/github_app/webhook references. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): skip deleted/archived workspaces in the auto-pull poller The poller scanned workspace_settings directly, so an archived (soft-deleted) or renamed-away workspace — whose settings row persists — kept polling and could enqueue a pull into a dead workspace. Join workspace and require NOT deleted. The EE webhook receiver gets the same filter (ee ref bumped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): never trust client-supplied server-owned auto-pull fields Both write endpoints (edit_git_sync_repository, edit_git_sync_config) persisted caller-supplied auto_pull.webhook_id / webhook_secret / webhook_error / last_synced_sha / last_pull_status when adding a repo or newly enabling auto-pull, letting a client inject a webhook id/secret or fake sync state. Strip those server-owned fields from the request up front; existing repos re-derive them from the DB (carried over), new ones start clean and the server (re)creates the webhook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): merge webhook fields post-commit instead of clobbering the row The post-commit webhook reconcile in edit_git_sync_repository and edit_git_sync_config wrote the whole pre-reconcile git_sync snapshot back after the main save committed. A concurrent git-sync edit or poller status write that landed in the gap could then be dropped by the stale snapshot. Re-read the current row and merge only the reconciled webhook id/secret/error for the repos the reconcile actually changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): parent-managed fork sync + PR-on-deploy toggles Fork sync (push-on-merge-to-forks parity): a parent-level auto_pull.sync_forks toggle routes changes on each fork's wm-fork/** branch into that fork workspace, via the parent's existing webhook and one extra fork-heads listing per poll tick (git ls-remote pattern for token repos, git/matching-refs for app-backed). Fork state is a server-written status-only auto_pull blob on the fork's own repo entry; the fork's card shows a read-only "managed in the parent workspace" line with its branch and last pull status. Dev workspaces (prefix-less ids) use the same branch parsing (unit-tested in windmill-common). PR-on-deploy: opening PRs for Windmill-pushed branches moves into the deploy pipeline, per repo toggle (promotion_open_prs on the promotion repo; parent-level fork_open_prs for fork deploys). The push job carries a marker and the job-completion hook derives the pushed branch (helper unit-tested against the CLI formula) and opens the PR outbound, so it works without inbound webhooks; the webhook-side wm_deploy PR arm is removed. The documented open-pr-* GitHub Actions remain valid alternatives (PR creation is idempotent). Fork guards: promotion mode, enabled auto-pull, and fork_open_prs are rejected on fork workspaces (they are parent-managed; a fork's deploys always target its wm-fork/** branch) and the promotion card is hidden in a fork's settings. Enabling auto-pull now also requires EE, and the post-commit webhook reconcile persists the normalized delivery mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): dev workspaces sync with their environment-label branch A dev workspace's git branch is its environment label verbatim (dev/ staging, default dev) — a first-class env branch like the documented push-on-merge-staging layout — instead of the wm-fork/** form. The label rides the deploy job args (backend → hub script → CLI --dev-workspace-label), the PR completion hook derives the same head, the webhook/poller route label branches into the matching dev-workspace child (poller lists them alongside wm-fork/* via extra ls-remote refs / per-label API lookups), and manual pulls from the UI pass clone_ref accordingly. The CLI refuses to deploy when the label branch equals the checked-out tracked branch, which would otherwise commit fork content straight to it. Because the branch is keyed on the label, the label is now immutable after creation: set at create/attach only, the set_dev_workspace_label endpoint is removed and the settings tab shows it read-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): nested fork routing + fork-of-dev branch rooting A fork of a dev workspace now roots its wm-fork/** branch on the dev's environment-label branch (the content it diverged from) and its PR merges back into that branch: the backend passes parent_dev_workspace_label with the deploy (parent row joined in both enqueue paths), the CLI gains --parent-dev-workspace-label and checks it before the wm-fork- prefix fallback when rooting a fork-of-a-fork branch, and the PR completion hook uses it as the PR base. Fork sync routing covers the whole live descendant chain of the webhook/poller workspace (recursive, depth-capped) instead of direct children only, and fork_open_prs is resolved at the root ancestor — only the root can hold auto-pull config, so grandchild forks sync through it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): PR deploy-preview comment, clearer check copy, app-only hints - The PR diff completion hook maintains one managed comment on the PR (Cloudflare deploy-preview style: workspace, status, commit, collapsible change list), upserted per synchronize via a hidden marker. The check run stays for required-check gating. - A settings difference in the diff summary is worded by cause: the PR changes wmill.yaml, vs pre-existing drift between the repo's wmill.yaml and the workspace, vs undetermined (neutral wording). - Deploy-status check titles name the target workspace ("Deployed 2 change(s) to staging"), since GitHub shows a head commit's checks on any PR containing it and a bare "Deployed" read as if the PR had deployed. - Token-based repos see a hint pointing at the open-pr-on-commit / open-pr-on-fork-commit workflows where the app-only PR toggles would be; an API-set toggle on a non-app repo now logs a warning naming the fallback; the design doc lists app-only features and their degradation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): EE-gate auto-pull UI, fork pull clone_ref, no-op push PR gate - CE: the auto-pull and fork-PR toggles are disabled with an EE badge, and new sync repos only default them on when licensed (basic git sync is available on CE since #8493, but auto-pull is EE and the backend rejects it) - The pull modal passes clone_ref for wm-fork- forks (wm-fork/<tracked>/<id>) so a manual pull fetches the fork branch instead of the tracked branch head - PR-on-deploy skips no-op pushes: when the push script reports pushed=false (e.g. the deploy was caused by an auto-pull), the completion hook no longer ensures a PR, so closed PRs aren't recreated by the sync loop Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * chore: refresh package-lock after main merge (windmill-utils-internal 1.8.2) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * test: auto-pull e2e integration tests; fix PR comment table formatting Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): runtime license gate for auto-pull saves; user/group promotion-branch parity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): explain in-sync PR verdicts with the repo's sync filter scope A PR that only touches files outside the repository's include paths gets "In sync", which reads as a wrong verdict; the check summary (and managed comment) now name the filters, e.g. "Only files matching this repository's sync filters deploy on merge: `f/**` (excluding `f/pat/**`)." Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): clearer card copy/structure; surface PR-creation failures - Fork sync toggle renamed and kept in the pull section; the fork PR toggle moves to the push section with a note that push settings apply to forks - Fork/dev workspaces' push section names their actual branch instead of the tracked-branch line; promotion repos hide the pull direction (promotion pushes deploy branches on top of a sync-mode setup) - Promotion mode line describes the wm_deploy/** branch + merge-to-promote flow; workflow-fallback hints lead with the how-to and link to the docs; test connection button demoted from accent per brand guidelines - New server-owned open_pr_error on repo settings: the deploy completion hook records why a PR couldn't be opened (e.g. app permission not yet approved) and clears it on the next success; shown as a warning under the PR toggles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix: cfg-gate scope-note helper (dead code on OSS builds) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): license-gate preserved auto-pull; attach strips parent-only settings - edit_git_sync_repository re-checks the runtime Enterprise gate against the EFFECTIVE repo state after preservation: the older-client arm copies the existing auto_pull back, which the request-side check never saw - attach_dev_workspace now mirrors the fork-creation copy on the attached workspace's own git sync: promotion repos dropped, auto_pull/fork PRs/PR error stripped, and any managed webhook deleted after commit (the attached workspace is parent-managed and must not keep pulling its old tracked branch) - integration test: attaching an auto-pull-enabled workspace strips it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): detach clears standalone parent; reject label == tracked branch - detach_dev_workspace clears parent_workspace_id for prefix-less (attached standalone) workspaces so they stop classifying as forks and deploying to wm-fork/** branches; wm-fork- re-designated forks keep their parent; cache invalidations mirror attach - dev-workspace create/attach reject an environment label that equals a git-sync repository's tracked branch (prod's or the candidate's): deploys would target the very branch the repo syncs from, and the CLI guard would fail every push job after the fact - CLI unit tests: prefix-less fork beats wm_deploy derivation; isForkWorkspace parent-id argument Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * chore(git-sync): bump hub script pins (push 28786, pull 28785) Published from windmill-integrations #163 with windmill-cli@1.753.1-gitsync.0: dev-workspace label deploys, fork-of-dev rooting, fork checkout on the existing remote branch, and the pushed-flag result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): no parent-only defaults on fork repos; rename strips webhook state - addSyncRepository skips the auto_pull/fork_open_prs defaults on fork/dev workspaces where the backend rejects them (saving a new sync repo from an EE fork 400'd deterministically) - change_workspace_id strips webhook id/secret/error from the copied git_sync and deletes the stale GitHub hooks post-commit: they deliver to the old (archived) workspace URL, so the new workspace would report a live webhook while polling at the relaxed interval; next save re-registers cleanly - EE: PR diff checks for contributor-fork PRs clone the synthetic pull/<n>/head ref (head.ref doesn't exist in the base repo) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * chore(git-sync): bump pull script pin to hub/28787 (synthetic PR ref support) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): targeted jsonb update for open_pr_error (no full-blob clobber) The full read-modify-write raced the poller's concurrent last_synced_sha / last_pull_status writes on the same column; mirror the EE status writer and update only the matching repository element's open_pr_error key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * style(git-sync): inline EE badge on gated toggles (matches settings nav) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * copy(git-sync): both directions in page/card descriptions; clearer promotion flow - Page header and sync-card description mention the pull direction, not only push-on-deploy - Promotion description walks the actual flow (wm_deploy/** branch, merge to promote, sync the target workspace) and points at the PR toggle / workflow; the Git Promotion docs link now also shows on configured cards, not only in the empty state Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): resolve branch-less resources' default branch for fork sync A git resource without an explicit branch polled as the bare "HEAD" ref, which the fork/dev-label fan-out cannot scope (wm-fork/<branch>/*), so fork sync silently never ran on polling-only repos. Resolve the remote's default branch name with `ls-remote --symref HEAD` (one call for name + head sha); "HEAD" only remains when resolution fails. The polling e2e test now uses a branch-less resource to cover this shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): runtime license gate for in-app PR creation promotion_open_prs/fork_open_prs are rejected on save without an Enterprise plan (like auto_pull), and the deploy completion hook re-checks the plan before opening PRs so flags stored while licensed stop driving GitHub calls after a lapse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): app-aware pull defaults, always webhook delivery, token-repo guidance - Pull-from-Git defaults on only for app-backed repos (applied when the selected resource resolves); polling is opt-in for token repositories, with a warning alert recommending the GitHub App (instant pull + in-app PRs) or the sync GitHub workflow - App repos always use webhook delivery with polling fallback: the delivery selector is gone and a stored polling mode is normalized back to auto - Post-save modal reflects the auto-pull state instead of telling the user to turn on a toggle that is already on - Non-app PR hints recommend the GitHub App explicitly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * copy(git-sync): single info box for token-repo pull guidance Merges the instant-pull recommendation with the GitHub Action conflict note, shown only for non-app repos; app repos need neither, and the redundant 'instant webhook sync requires' line is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * copy(git-sync): keep the GitHub Action conflict note on app repos Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * copy(git-sync): anchor docs links to their exact sections GitHub App references point at integrations/git_repository#github-app, the workflow hints at deploy_gh_gl#github-actions-setup, and the sync workflow at git_sync#github-actions (all anchors verified against the live docs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * copy(git-sync): fork workflow hint links to git_sync#github-actions open-pr-on-fork-commit is documented on the git_sync page, not deploy_gh_gl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * fix(git-sync): fork PRs are opt-in on new connections too Only auto-pull and fork sync default on for new app-backed connections; opening pull requests stays a deliberate per-repo decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * feat(git-sync): default the managed PR on for new app-backed promotion repos A promotion deploy's wm_deploy/** branch exists to be merged; without a PR it's an orphaned branch. Fork PRs stay opt-in. Also scope the sync-repo auto-pull default to sync mode so promotion repos can't pick it up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * docs(git-sync): GHES self-managed app permission setup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * docs(git-sync): frame permission update against GitHub Actions, not polling Existing installations don't have polling; their git-to-Windmill direction runs on GitHub Actions today, so the approval text describes the update as replacing those workflows and notes every feature is opt-in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm * copy(workspaces): drop 'cosmetic' qualifier from dev-workspace label UI * chore: update ee-repo-ref to 9b2a6375f838436cf68cff449cc9bc621cca5281 This commit updates the EE repository reference after PR #632 was merged in windmill-ee-private. Previous ee-repo-ref: 99eef24e2f0402b9a997cde5f67be52ee5d54b0e New ee-repo-ref: 9b2a6375f838436cf68cff449cc9bc621cca5281 Automated by sync-ee-ref workflow. * fix(git-sync): reject '/' in fork and dev workspace ids * fix(git-sync): bound auto-pull git probes with a per-command timeout * fix(git-sync): persist webhook reconcile via targeted jsonb updates --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-16 15:55:44 +02:00
3 people	3bd9f05938	fix(alerts): identify server replica in low-disk alert + per-host dedup tag (#10143) ... * fix(alerts): identify server replica in low-disk alert + per-host dedup tag The server-mode low-disk alert keyed its dedup tag on the mountpoint alone, so `simple_alert_helper` mapped every server replica onto a single alert row per mountpoint. With more than one replica that row flaps every monitor pass: a replica seeing low disk raises the alert while a replica seeing healthy disk recovers it. The alert text also could not say which replica tripped. The fix lives in windmill-ee-private (`low_disk_alerts` in windmill-common/src/ee.rs) and appends the hostname to both the message and the dedup tag, mirroring the worker branch. Also add a regression test pinning the server tag as per-host, and correct the monitor cadence comments: iterations are LISTEN_NEW_EVENTS_INTERVAL_SEC (10s by default), not 30s, so "~60s (2 iterations * 30s)" was wrong on both factors. * fix(alerts): widen healthchecks.check_type so per-host disk tags fit Alert tags embed a mountpoint and a hostname, both unbounded, but check_type was varchar(50). create_alert only logs the insert error while the notification still fires, so an overflowing tag re-alerts every monitor pass and never records recovery state. The server tag overflows for ordinary pod-length hostnames, and the existing worker tag already overflows for every tracked mount except "/". Widening the column fixes both; bounding the hostname would not, since the mountpoint alone can consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 This commit updates the EE repository reference after PR #666 was merged in windmill-ee-private. Previous ee-repo-ref: ccd1e42cf6b2d051ca17074fbdf5b80a46cffe0f New ee-repo-ref: b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-16 15:21:54 +02:00
Ruben FiszelandClaude Opus 4.8	f7eb5c460d	fix(apps): invalidate cached app policy on change or deletion (GHSA-r5v4-cxh9-7qhq) (#10121) ... * fix(apps): invalidate cached app policy on change or deletion (GHSA-r5v4-cxh9-7qhq) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(agents): keep PR tests and comments minimal and non-ephemeral Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:41:54 +02:00
3 people	ff774c46bf	feat: add per-workspace job-retention override (#10050) ... * feat: add per-workspace job-retention override (EE) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 2ba6a2a75b6fc97858b306b2c98ada481e363c10 This commit updates the EE repository reference after PR #658 was merged in windmill-ee-private. Previous ee-repo-ref: e7fb36acd813cd717bcf05f5aafbf81de271d618 New ee-repo-ref: 2ba6a2a75b6fc97858b306b2c98ada481e363c10 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-11 23:46:34 +02:00
Ruben FiszelandClaude Opus 4.8	f28ea9cb99	feat(db-health): add connection sizing guidance (#10014) ... * feat(db-health): add connection sizing guidance The Database Connections panel showed current/max connections but gave no guidance on how to size max_connections for the deployment. Derive an estimate from the live worker fleet: each worker instance shares a pool sized DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1), and each server opens up to DEFAULT_MAX_CONNECTIONS_SERVER (both overridable via DATABASE_CONNECTIONS). The endpoint now returns live worker/instance counts, the default per-server and per-worker pool sizes, the estimated peak worker connections, the reserved superuser connections, and a recommended max_connections floor (workers + one server + 25% headroom). Servers do not ping worker_ping, so the recommendation assumes one server and exposes the per-server increment. The panel renders this as a sizing breakdown and warns when max_connections is below the recommended floor. Fixes WIN-2147 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(db-health): single source for pool-size constants + sizing tests Address review: db_connect.rs kept its own copies of DEFAULT_MAX_CONNECTIONS_* that duplicate the windmill_common constants the sizing guidance reads, so tuning the runtime pool size would silently leave the guidance stale. Re-export the windmill_common constants from db_connect.rs so there is one source of truth. Add unit tests for compute_connection_sizing covering the zero-fleet, single worker, multi-instance, and reserved-clamp cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-health): 20% headroom and 200-connection minimum floor Lower the sizing headroom from 25% to 20% and never recommend below 200 connections (postgres defaults to 100; cheap headroom for growth/bursts/psql). Update the guidance message and unit tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): honor DATABASE_CONNECTIONS in sizing recommendation Address Codex P1: the runtime caps every process's pool at DATABASE_CONNECTIONS when set (db_connect.rs), but the sizing guidance always used the default 50/5 pools. For a tuned deployment this under-estimated worker demand and could hide a genuine under-provisioning (e.g. DATABASE_CONNECTIONS=100 with 5 instances is 500 worker connections, not 25). compute_connection_sizing now takes the effective DATABASE_CONNECTIONS override (read the same way db_connect.rs reads it): when set, each worker instance and server pool is that value and the worker estimate is override * instances. The response exposes server_pool_size / worker_pool_size (effective) and database_connections_override; the panel renders both pool rows and labels them (default) vs (DATABASE_CONNECTIONS), and the message states which source is used. Adds a unit test for the override path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): exclude agent workers from connection sizing Agent workers reach the API over HTTP (MODE=agent, Connection::Http) and hold no postgres pool, but their pings still land in worker_ping (written server-side by /api/agent_workers/update_ping). Counting them inflated the connection estimate. Filter the fleet query by the worker-name prefixes: DB-connected workers use "wk-" (WORKER_NAME_PREFIX), agent workers use "ag-" (AGENT_WORKER_NAME_PREFIX). Only wk- workers/instances feed the estimate; ag- workers are counted separately and surfaced as context ("N agent workers excluded — they use HTTP, not postgres connections"). Adds a unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-08 16:42:39 +00:00
Ruben FiszelandClaude Opus 4.8	91e1b087a2	feat(auth): add runtime NO_AUTH mode for authentication bypass (#9962) ... * feat(auth): add runtime NO_AUTH mode for authentication bypass Adds a runtime `NO_AUTH` env flag that makes every request resolve as the `admin@windmill.dev` superadmin with no login required, so self-hosted deployments can front Windmill with their own authenticating gateway without building a dedicated `oss` (compile-time `no_auth`) binary. - `NO_AUTH` is honored in any build but is force-disabled when `CLOUD_HOSTED` is set, so the managed cloud always enforces real auth. - The existing compile-time `no_auth` feature keeps its always-on behavior (`cfg!(feature = "no_auth") || *NO_AUTH`), so `oss` builds are unchanged. - `Tokened` now yields a synthetic token in no-auth mode so handlers that require it (e.g. global_whoami, called by the frontend on load) resolve. - A loud startup banner warns when the mode is on; `HIDE_NO_AUTH_BANNER` silences it once the operator has deliberately deployed behind a gateway. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(auth): dismissable NO_AUTH warning banner via global setting Replaces the HIDE_NO_AUTH_BANNER env flag with a UI warning banner that can be permanently dismissed for all users from within the running instance (not exposed in instance settings). - New `no_auth_banner_dismissed` global setting, only ever written by dismissing the banner itself. - `GET /api/settings/no_auth_banner` returns whether to show the banner (true only when NO_AUTH is active and it hasn't been dismissed). - NoAuthBanner.svelte renders a top-of-app warning in NO_AUTH mode; its dismiss button opens a confirmation modal, then writes the global setting via the existing setGlobal endpoint so it stays hidden for everyone. - The server still logs the startup NO_AUTH warning unconditionally. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(auth): resolve NO_AUTH in AuthCache so all_runnables works Codex/Pi review flagged that `/api/users/all_runnables` still failed in NO_AUTH mode: `get_all_runnables` extracts `Tokened` and re-validates the request token per workspace via `AuthCache::get_authed`, which rejected the fabricated `"no_auth"` token (no matching DB row) with a 400. Short-circuit `AuthCache::get_opt_job_authed` (the resolver behind `get_authed`) to the admin superadmin in no-auth mode, so any direct cache caller resolves without a real token. Single-source the mode check and the synthetic identity via `is_no_auth()` / `no_auth_admin_authed()` and reuse them across the extractor, resolver, and login paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(auth): drop the NO_AUTH dismissable UI banner The in-app banner added a GET /api/settings/no_auth_banner request to every instance load for little benefit. The startup log warning already surfaces that auth is bypassed to operators, so drop the banner, its endpoint, and the no_auth_banner_dismissed global setting entirely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-06 17:02:51 +00:00
3 people	5d7fb6deca	feat(pipelines): asset freshness — fresh/stale badge (CE) + watchdog (EE) (#9909) ... * feat(pipelines): passive asset freshness tracking on the graph Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(pipelines): drop dead freshness-enforcement stub, document query ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): freshness watchdog (EE) — auto re-run stale producers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): watchdog review fixes — archived workspaces, badge kind parity, scan index Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — no singlestepflow in freshness, +N parity, completion-time fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — history completedAt, freshness/asset trigger UI metadata Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6f5fe0f7f56696fbef5a8349da38496c32e71666 This commit updates the EE repository reference after PR #643 was merged in windmill-ee-private. Previous ee-repo-ref: 1f13380354bf591ae25a2c20d36917534bcc5459 New ee-repo-ref: 6f5fe0f7f56696fbef5a8349da38496c32e71666 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-04 06:23:44 +02:00
Ruben Fiszel	84141add1d	feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890) ... * feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud	2026-07-03 00:59:30 +02:00
Ruben FiszelandClaude Fable 5	d15033cde6	fix: invalidate bun bundle cache on transitive relative-import changes (#9891) ... * fix: invalidate bun bundle cache on transitive relative-import changes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: do not memoize transient fetch errors in bundle-key import cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use regular comment on lazy_static block (deny unused_doc_comments) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: align bundle-key import version selection with loader content endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 23:57:26 +02:00
3 people	5549bdc67a	fix(debounce): never supersede a running debounce survivor (#9780) ... * fix(debounce): never supersede a running debounce survivor Companion to the windmill-ee-private change in upsert_debounce_key. With debounce_args_to_accumulate + a concurrent_limit, a message arriving while its debounce survivor is already running was marked completed/skipped ("Debounced Running by ...") and the running survivor deleted from the queue, silently dropping accumulated elements. A slow step + concurrent limit keeps the survivor running for a long window, so any arrival during it was lost. The fix leaves a running survivor untouched and starts a fresh debounce window for the late arrival. Adds regression coverage in windmill-queue/tests/debounce_test.rs (push, flow post-preprocessing, no-accumulation, committed-running, and max-count-window cases) and refreshes the SQLx cache for the changed upsert_debounce_key queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(debounce): add missing SQLx cache for test-only running-flag query The cargo_test CI job compiles the test target with SQLX_OFFLINE=true; the new regression tests use `UPDATE v2_job_queue SET running = true ...` which was not in the offline cache (the library-only `cargo sqlx prepare` skipped test targets). check_oss/check_ee passed because they don't build tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(debounce): harden running-survivor guard against concurrent arrivals Companion to windmill-ee-private: switch the running-state check to a correlated EXISTS on the post-conflict-lock holder so two late arrivals racing after a survivor started running can't both spawn independent windows (the row lock serializes them; the second debounces into the first's fresh window). Adds a concurrent regression test (test_debounce_concurrent_arrivals_after_running_survivor) asserting exactly one late arrival survives and the other is debounced, and refreshes the SQLx cache for the updated upsert_debounce_key queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(debounce): serialize upsert per key (simpler, race-free) Companion to windmill-ee-private: the running-survivor guard and batch chaining are now protected by a per-key advisory lock instead of snapshot-sensitive single-statement SQL. This closes a concurrent-arrival data-loss race where a debounced late arrival's args could be dropped because the batch lookup couldn't see the predecessor's just-committed batch row. Extends test_debounce_concurrent_arrivals_after_running_survivor to pull the survivor and assert its accumulation includes BOTH racing late arrivals (shared batch), and refreshes the SQLx cache for the rewritten queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(debounce): atomic upsert robust to concurrent pull-time key deletion Companion to windmill-ee-private: keep upsert_debounce_key a single atomic INSERT ... ON CONFLICT DO UPDATE so a chaining push cannot fail when the worker pull path concurrently deletes the holder's debounce_key (the prior read+UPDATE split could hit "no row updated"). Adds test_debounce_push_races_key_deletion_by_pull (races a chaining push against the key deletion 50x, asserts the push never errors) and refreshes the SQLx cache. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(debounce): claim-based exactly-once batch consumption Eliminates the rare duplicate/loss when two survivors land on one debounce batch (a narrow push/pull race), without locking the worker pull hot path. - migration: v2_job_debounce_batch gains consumed_at + consumed_by. - pull side (maybe_apply_debouncing): instead of deleting the batch on consume, a survivor atomically claims its own row + any unclaimed siblings (stamping consumed_by = itself) and accumulates exactly the rows it claimed. A second survivor of the same batch finds its row already consumed by another job and runs empty (no duplicate); a re-pulled survivor recognizes its own prior claim and keeps its accumulated args; a never-batched job (CE/legacy) keeps its own args. Non-accumulate debounce paths still hard-delete their batch rows. - complete_debounced_job (EE companion) never completes a running predecessor, so its in-flight run is not killed (no loss); the claim then prevents the duplicate the guard would otherwise allow. - monitor: GC sweep deletes consumed batch rows past a 1h grace. Together with the running-survivor guard this makes debounce accumulation exactly-once. Adds tests: batch_consumed_exactly_once, repull_keeps_accumulated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(debounce): exhaustive edge cases + tighten consumed-batch GC grace Tighten the consumed debounce-batch GC grace 1h -> 10min: per-op cost of the claim is unchanged (an indexed mark is as cheap as the old delete), so the only cost of retaining consumed rows is table growth, which a shorter grace bounds under high-throughput debounce (a survivor that could still reference a row is pulled long before 10min; GC is not correctness-critical since a re-pull whose row was swept falls back to its persisted args). Adds edge-case tests: never-batched keeps own args (CE fallback), concurrent claim partitions a batch disjointly (exactly-once under real concurrency), three survivors -> first takes all / rest run empty, non-accumulate debounce hard-deletes its batch rows (no leak), and the GC sweep deletes only past-grace consumed rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(debounce): port the #9781 regression case, flow-node guard, full-path bench - Port the regression from #9781 (test_post_preprocessing_debounce_into_running_survivor_loses_message): post-preprocessing survivor accumulates + runs, a later same-key message must start a new batch (survive) not be folded into the running survivor. Exercises the full EE path via jobs_ee::maybe_debounce_post_preprocessing. - Add the third EE entry point's guard: test_flow_node_debounce_running_survivor_not_superseded (maybe_debounce_flow_node). - Add an #[ignore] full-source throughput bench (bench_debounce_full_path) driving the real maybe_debounce + maybe_apply_debouncing end-to-end. All debounce tests exercise the real jobs_ee implementation (run with --features private,enterprise); none stub it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(debounce): scalar-arg accumulation + GC-then-repull no-loss Close two accumulation edge gaps (both run on --features private,enterprise, exercising the real jobs_ee path): - accumulate bare-scalar values (the T | T[] union fallback): each scalar is wrapped and accumulated into the survivor's list. - GC reclaiming a survivor's consumed batch row before a re-pull must not lose data: the re-pull finds no row and keeps its already-persisted accumulated args (had_row=false fallback), rather than running empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(debounce): real-worker end-to-end accumulation test Drives the full real path on --features enterprise,deno_core,private: push 3 same-key debounced flow jobs (real push() -> maybe_debounce collapses the batch), a real worker pulls the survivor (real pull() -> maybe_apply_debouncing claim+accumulate) and executes the deno flow, then asserts the executed result is the full accumulated set [1,2,3] and the two superseded messages are skipped. Complements the in-process unit tests with a genuine worker-execution run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(debounce): atomic claim+persist, GC only non-queued rows; reword comment Address review findings: - [P1] Claim and accumulated-args persist are now in one transaction. Before, a crash between stamping batch rows consumed_by=self and the `UPDATE v2_job SET args` could let a zombie re-pull see its own prior claim and keep only its own args (dropping the siblings it had claimed). Wrapping claim + accumulate + persist in a tx makes them commit together or roll back together (re-pull then re-claims cleanly). - [P1] GC of consumed batch rows now also requires the job to no longer be in v2_job_queue. A consumed sibling can stay queued well past any time grace under a concurrency limit / backlog; reclaiming its marker by age alone let its eventual pull treat it as never-batched and re-run its item (a duplicate). Keeping the row until the job leaves the queue preserves the "already consumed" signal. Test extended with a still-queued consumed row that must survive GC. - [P2] Drop "Customer" attribution from a test doc comment (AGENTS.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(debounce): emit accumulation log after committing the claim transaction append_logs opened a second pool connection while the claim transaction (and its batch row locks) were still held; under concurrent debounced pulls that risks pool-exhaustion stalls/timeouts. Defer the log line until after tx.commit(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a This commit updates the EE repository reference after PR #631 was merged in windmill-ee-private. Previous ee-repo-ref: 30d740e619fad219108ec4b4c6a9d67c1ab42d46 New ee-repo-ref: 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a Automated by sync-ee-ref workflow. * fix(debounce): claim whole batch in one UPDATE (no deadlock); assert test setup Both Codex (P1) and Claude (P2) flagged a deadlock: the claim used two writable CTEs (claim_self then claim_rest), locking the self row before siblings, so two survivors of the same batch pulled concurrently acquired row locks in opposite order and PostgreSQL aborted one with deadlock_detected (a transient pull error on exactly the two-survivors race this path handles). Replace with a single `UPDATE ... WHERE debounce_batch = (...) AND consumed_at IS NULL RETURNING id` that claims the whole batch: both transactions lock rows in the same scan order, so one simply waits and re-evaluates under EvalPlanQual. A `claimed_self` flag (EXISTS id = self in the claimed set) plus the `mine` snapshot still distinguishes fresh-claim / consumed-by-other / own-re-pull. Also assert add_survivor_to_batch_of actually inserts a row (rows_affected == 1) so a mis-set-up test can't pass vacuously. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-06-25 16:36:16 +00:00
Ruben FiszelandClaude Opus 4.8	aa098c70c0	perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes (#9786) ... * perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: document delete_jobs auth contract and workspace-scope jobs_export purge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 13:52:30 +00:00
Ruben FiszelandClaude Opus 4.8	12f92e3ab7	[ee] feat(backend): native script retry without one-step-flow wrapping (#9688) ... * feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 13:22:26 +00:00
Ruben FiszelandClaude Opus 4.8	8912e21d15	perf(monitor): vacuum job_perms/job_result_stream right after each orphan sweep (#9753) ... A customer's top-load query was the job_perms orphan sweep (cleanup_job_perms_orphaned: 6.9s mean, 41s max). The cost is discovery, not deletion (~2.1ms per row deleted): the NOT EXISTS anti-join seq-scans the whole job_perms heap to find a few orphans, and that scan tracks the heap's physical size. job_perms / job_result_stream_v2 get one row per job and are drained only by these per-cycle sweeps, so they churn hard — but the bulk vacuuming_tables() runs only ~hourly, so dead tuples bloat the heap between bulk vacuums. Reclaim right after each sweep instead: VACUUM (SKIP_LOCKED) the swept table when it deleted rows. Plain VACUUM (not FULL) takes only SHARE UPDATE EXCLUSIVE so concurrent job creates/reads proceed; the visibility map skips unchanged pages so repeated runs are cheap; SKIP_LOCKED means HA replicas don't pile up (one vacuums, the rest skip). Benchmarked ~7x: a bloated 268MB job_perms heap swept in 35ms vs 5ms vacuumed. Chosen over an autovacuum reloptions migration so the behavior is explicit and lives with the sweep it pairs with. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-24 10:20:53 +02:00