mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
Ruben Fiszel
30f5d2e766
* perf: read global_settings once per settings-load pass `initial_load` reads several dozen settings back to back, one `SELECT value FROM global_settings WHERE name = $1` each: 50 serialized round trips before a worker is ready, 32 before a server is. On localhost that is ~20ms and invisible; against a real database it is 50x the RTT per process start, which `EXIT_AFTER_N_JOBS` turns into a per-job cost. `with_global_settings_snapshot` reads the whole table (12 rows on a typical instance) into a tokio task-local, and `load_value_from_global_settings` serves from it. Scoping it to the task is what keeps the single-setting reload paths correct: a `notify_global_setting_change` event for one key runs outside any scope and still reads the database, so a live settings change reaches a running worker as before. Agent workers hold an HTTP connection with no snapshot to take and are unchanged. `load_smtp_config` and `reload_custom_tags_setting` had their own inline copies of the same query; they go through the shared loader so they land in the snapshot too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the snapshot contract on the reader and the query `load_value_from_global_settings` is called from ~10 crates and one of them writes a setting then immediately re-reads it through `reload_custom_tags_setting`; say on the function itself that a scope, when one is installed, serves the read and leaves `db` unused. The query comment claimed the table is a handful of rows. It is not bounded that way: `workspace_dependencies_map_rebuilt:<workspace_id>` adds a row per workspace and never removes it. Those dynamically named rows are also why the snapshot fetches the whole table instead of the wanted names, so state that as the reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound the settings snapshot and keep it out of two reads Three review findings, all real: The snapshot fetched the whole table, which is not bounded by the settings that exist: `workspace_dependencies_map_rebuilt:<workspace_id>` adds a row per workspace with no cleanup path, and no settings pass reads one. It now fetches only statically named rows, and reads of a `<prefix>:<id>` name skip the snapshot and go to the database. Correctness does not rest on that naming convention — a colon-free dynamic name would simply be in the snapshot and still answered correctly — only the bound does. A snapshot query that failed inside an enclosing snapshot awaited the body bare, so its reads were served by the outer snapshot rather than falling through as documented. The task-local carries an explicit bypass state and the failure path scopes it. `reload_jwt_secret_setting` decided whether to generate-and-upsert the JWT secret from a snapshot-served read, so a replica booting alongside another could overwrite the secret it had just generated and invalidate its tokens. That read goes through the new `load_value_from_global_settings_fresh`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the snapshot query on the primary-key index `name NOT LIKE '%:%'` bounded the rows returned but not the work: a leading wildcard cannot use the index, so Postgres read every row anyway. Against 50k dynamically named rows it plans as a seq scan of 516 buffers whether or not seqscans are enabled — and worker connections disable them, so the plan was one the query shape forbade rather than one the planner chose. `name = ANY($1)` over an explicit list plans as a bitmap index scan, 7 buffers, bounded by the listed names rather than by table size. That list is also exactly the set the snapshot may answer from, so a name outside it falls through to the database instead of reading as unset: listing a setting is a performance choice, never a correctness one, which is what keeps the list safe to maintain by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: declare a settings pass instead of reading one setting at a time Replaces the prefetch-list snapshot with a pass the call sites build themselves. `SettingsPass` collects the reads `initial_load` will make as `(name, applier)` pairs, fetches them together, then replays the appliers in declaration order. Declaring is what makes the batch exact. The same `if server_mode` / `if *CLOUD_HOSTED` / `cfg` branches that used to guard a read now guard a declaration, so the fetch asks for what this process needs and nothing else, and there is no list of setting names to keep in sync with anything. Ordering is preserved end to end: appliers run in the order they were declared, and non-setting work in the middle of the sequence keeps its place as a step, so nothing moves and nothing runs twice. Steps that need several settings at once take them together. The batch distinguishes three states where a per-setting read only ever produced two at a given call site: - a value, - genuinely unset, which several settings must see in order to restore a default when the setting is cleared, - could not be read, which must leave the in-memory value alone. Collapsing this into "unset" would let one failed query reset workspace fairness and the queue caps across a cluster. Over HTTP the reads go out together rather than sequentially, so an agent worker's settings load costs one round instead of ~36, with no new endpoint. A setting an agent may not request still resolves to unset, as the per-setting call returned for it. `reload_*` keeps working per setting for the notify path, sharing its apply half with the pass. The wrappers no caller was left using are dropped. worker startup: 50 queries -> 2 (the batch, and jwt_secret which stays its own read so the pass cannot sit between reading it absent and upserting a replacement over another replica's). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: run the pass's non-setting steps in declaration order too Review round found the settings pass had a gap: the reads were declared but the work interleaved between them still awaited inline, so it all ran before `pass.run` applied anything. `manage_audit_partitions` therefore saw `AUDIT_LOG_RETENTION_DAYS` at its compile-time default rather than the configured value, and dropped every partition past that default. An instance keeping 30 days on CE lost the 14-to-30-day band on startup and on every full-reload tick. The `STORE_AUDIT_LOGS_S3` export anchor had the same cause: the gate read `false` before the setting applied, so an env-var-enabled export never anchored and its first tick skipped the rows committed before it. `action` exists so a step keeps its place in the sequence; every remaining inline await is now one, which fixes both and leaves no phase where a read can observe a value the pass has not applied yet. Two more from the same round: A batch that fails as a whole now falls back to per-setting reads. Skipping every applier preserves known-good state on a reload tick, but a starting process has none, and would have run on compile-time defaults until the next full reload twelve hours later. `FORCE_RUBY_REPOS` is honored again: the batched url-list path parsed without the `FORCE_` check its per-setting counterpart applied, so the override was silently dropped. `load_setting_value` never had one, so the third helper was never affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: declare the object-store and worker-config steps in the pass too Two awaits were left running ahead of `pass.run`, so the settings they read were still at their compile-time defaults. The object-store reload is the one that matters: an AWS OIDC store mints its first token against an issuer built from `BASE_URL` (`oidc_ee.rs`), and with `OTEL_ENVIRONMENT` set nothing loads that before this pass does, so the store signed with the unset default, left `OBJECT_STORE_SETTINGS` empty and fell back to the ten-second retry while startup carried on. `reload_worker_config` calls `store_pull_query`, which reads the workspace fairness knobs. It happened to converge because the enabled flag re-stores the query when it changes, but it was reading defaults on the way there. Both are steps now, which is also what the earlier fix should have covered: the only await left outside a step is `pass.run` itself. Also from the same round: `fetch_settings_batch`'s doc comment had been stranded on the helper inserted above it, and the batch-failure fallback re-ran the same reads on an agent worker, where the batch already is the per-setting read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: point the setting-loader docs at functions that still exist `reload_setting` went with the other wrappers no caller was left using, but two doc links still referenced it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: decide the jwt secret in sql so the read can be batched `reload_jwt_secret_setting` generated a secret whenever its read came back absent or unparseable, and upserted it unconditionally. Two replicas booting against an empty row therefore each installed their own and rejected each other's tokens, and the same happened on a running cluster whenever the row was deleted or set to a non-string. Keeping the read next to the write kept the window narrow but never closed it, and it was the reason this one setting could not go through the settings pass. `get_or_create_jwt_secret` puts the decision in the statement instead: INSERT ... ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value WHERE jsonb_typeof(global_settings.value) <> 'string' RETURNING value First writer wins, a usable secret is never overwritten, and an empty RETURNING is how a caller learns another process's secret stands. The `WHERE` also keeps a normal startup from writing at all, which matters because `notify_global_setting_change` fires on every write to this table and an unconditional upsert would have made each start trigger a cluster-wide reload. Because the statement decides rather than the caller's read, a stale value is harmless and `jwt_secret` is now an ordinary declaration. Worker startup is a single batch round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a failed read from dropping a FORCE_ override or clearing a setting Two ways a read that did not succeed was being treated as an answer. A `FORCE_` override used to be checked before the read, so a failed read could not affect it. Moving that check into the parser put it behind a value arriving, and a failed read skips its applier, so a forced private registry fell back to the public index and a forced `settings.xml` was deleted from disk by the Maven step that follows it. Forced settings are declared as steps with no read now: the override outranks the database, so there is nothing to fetch and nothing to lose when a fetch fails. The setting loaders were passing `v.ok().flatten()` to their appliers, which turns a database error into "unset". Most appliers ignore `None`, but `apply_tag_per_workspace_workspaces` clears the workspace whitelist with it, making every workspace eligible for per-workspace tags, and `apply_fork_workspace_tag_append_fork_suffix` stores `false`. Both are also reached from the notify handlers, so a blip during a reload changed routing for the cluster. They take `?` now, as the code they replaced did by leaving the error arm empty, and the other five are converted with them so an applier that later grows a `None` branch cannot inherit the problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: route hub_api_secret through the FORCE-aware declaration `HUB_API_SECRET` lives in an `ArcSwap` rather than an `Arc<RwLock<_>>`, so it could not use `option_setting` and was declared by hand with a bare `setting` plus `parse_option_setting_value` — which is exactly the path that skips the `FORCE_` handling, so a failed read still dropped `FORCE_HUB_API_SECRET`. The rule now lives in `option_setting_with`, which takes the store closure and leaves `option_setting` a wrapper over it, so a setting held in something other than an `RwLock` reaches it too rather than having to reimplement it. The three remaining hand-written parses are `parse_setting_value`, which has no `FORCE_` handling to miss: `load_setting_value` never had the check either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>