Python jobs under nsjail experience slow imports (~465ms for `import
requests`) because dependency directories are mounted read-only. Without
pre-compiled `.pyc` files, Python recompiles `.py` source to bytecode
in-memory on every import in every fresh nsjail process, paying the cost
repeatedly.
Add `--compile-bytecode` to both `uv pip install` invocations (the
Rust-driven install in python_executor.rs and the nsjail
download_deps.py.sh script) so `.pyc` files are generated at install
time and available at runtime even through read-only mounts. The files
are included in both the local cache and S3 piptar uploads.
The flag is supported by the uv version (0.9.24+) shipped in the
Dockerfile. The existing `__pycache__` skip only filters top-level dirs
for dedup logic, not subdirs within packages, so there is no conflict.
Fixes WIN-2001
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(api): handle multi-version scripts when removing granular ACL
remove_granular_acl built a CTE whose `old` subquery selects one row per
matching table row, then read it back via the scalar subquery
`(SELECT old_write FROM old)` in the RETURNING clause. For the `script`
table the PK is (workspace_id, hash), so a path with multiple deployed
versions yields several rows sharing the same (workspace_id, path). When
two or more versions carried the ACL key, `old` returned multiple rows and
PostgreSQL rejected the scalar subquery with "more than one row returned by
a subquery used as an expression", making it impossible to remove an ACL
entry from a script's permissions panel.
All versions share the same extra_perms value (the UPDATE applies to every
matching row), so any single row's old_write is representative. Add
`LIMIT 1` to the scalar subquery. Other kinds are unaffected because they
have a unique constraint on (workspace_id, path/name).
Introduced by b3603d872 (#7365). Add a regression test reproducing the
multi-version case at the SQL level.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: remove granular ACL regression test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim ACL fix comment
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cache): allow overriding hub base url via env in `cache` mode
The `windmill cache hubPaths.json` prebuild step (run in the Dockerfile) never
connects to the DB, so HUB_BASE_URL stays at its compiled default
(https://hub.windmill.dev) — unlike server/worker modes which load it from the
DB global setting. This made it impossible to point the prebuild cache step at
a private or staging hub.
Read HUB_BASE_URL from the environment at the start of cache_hub_scripts and
store it into the existing HUB_BASE_URL ArcSwap (the same static the hub fetch
functions read). No effect unless the env var is set and non-empty; server and
worker modes are unchanged (they still use the DB setting).
This also enables validating hub-script dependency changes end-to-end against a
local fake-hub before pushing to the real hub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): re-pin cached hub scripts to patched versions
windmill-integrations#133 was merged and pushed to the hub, minting new
versions with regenerated (CVE-free) lockfiles. Bump the hubPaths.json pins so
the prebuild cache step (`windmill cache`) fetches the patched lockfiles instead
of the old vulnerable ones (the hub serves each version_id immutably, so the old
pins keep returning the vulnerable deps until bumped).
- slackErrorHandler 19741 -> 28241
- slackRecoveryHandler 9080 -> 28239
- slackSuccessHandler 28220 -> 28240
- smtpReport 9086 -> 28242
- appReport 28076 -> 28243 (puppeteer screenshot script)
- gitInitRepo 28219 -> 28229 (already-fixed hub version; pin was stale)
Validated end-to-end against the real hub: `windmill cache` with these pins
produces a clean cache_nomount/bun (axios 1.16.1, form-data 4.0.5,
follow-redirects 1.16.0, nodemailer 8.0.10, ws 8.21.0, svelte 5.55.8,
devalue 5.8.1; basic-ftp and ip-address no longer pulled). No vulnerable
versions remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flows): preserve step/subflow worker tags under a custom-tagged flow
A flow running on a custom worker tag force-propagates that tag to every
descendant step, script and nested sub-flow, overriding their own declared
tags. This made it impossible to route a specific step or sub-flow to a
different worker group. The new opt-in FlowValue.preserve_step_tags lets a
step that declares its own non-empty tag run on it; untagged steps still
inherit the flow tag. Defaults off to preserve existing behavior.
* chore: regenerate system prompts for preserve_step_tags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(flows): nest preserve_step_tags toggle under flow worker tag setting
The toggle only affects routing when the flow has a custom worker tag, so
show it as a sub-setting of the Worker Group tag picker, visible only once a
tag is set, instead of as a standalone option.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): allow step worker tag picker when preserve_step_tags is enabled
When a flow defines a worker tag, the per-step tag picker was replaced by a
read-only "Flow's WG" label. With preserve_step_tags enabled the step's own
tag is honored, so the picker must remain editable in that case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): propagate preserve_step_tags to branch and loop bodies
payload_from_modules built the synthetic RawFlow for branch/loop bodies with
a default FlowValue, dropping preserve_step_tags. Tagged steps inside a
branch or loop therefore still inherited the parent flow tag even with the
flag enabled. Thread the flag through to the synthetic FlowValue so the
behavior is consistent for nested containers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): clear preserve_step_tags when flow worker tag is removed
Avoids the flag lingering as invisible state after the flow tag (and its
toggle) are removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): repair preserve_step_tags propagation to branch/loop bodies
The previous commit added flow.preserve_step_tags at the payload_from_modules
call sites but the parameter and FlowValue field were not actually threaded
through (a failed edit left the function unchanged), so the crate did not
compile. This completes the change: payload_from_modules takes preserve_step_tags
and sets it on the synthetic FlowValue for branch/loop bodies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): complete preserve_step_tags propagation to branch/loop bodies
Previous two commits left windmill-worker uncompilable: payload_from_modules
received flow.preserve_step_tags at its call sites but the parameter and the
synthetic FlowValue field were not actually added. This adds the parameter,
sets preserve_step_tags on the synthetic FlowValue, and threads
flow.preserve_step_tags through all five call sites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): clear preserve_step_tags whenever the flow worker tag is removed
The flag was only reset when the Worker Group toggle was switched off, not
when the tag was cleared directly in the picker (or via the YAML editor),
leaving preserve_step_tags=true as invisible state with the advanced badge
still reporting it active. Move the cleanup into the reactive block that
already tracks the flow tag so every clear path is covered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(apps): make public apps opt into cross-origin isolation via wm_coep
Public app pages served at /public/* and custom paths /a/* were not
getting the COEP/COOP/CORP headers, so they were blocked when embedded
as an iframe inside a cross-origin-isolated page (e.g. another raw app,
which sets Cross-Origin-Embedder-Policy: require-corp). A nested
document loaded into a require-corp context must itself set COEP for
the iframe to load.
Rather than applying the isolation headers to all public pages (which
would also force COEP on classic apps and break subresources without
CORP, e.g. external image URLs or embeds), public apps now opt in via
a `wm_coep` query param on the embed URL:
<iframe src="https://<domain>/public/<ws>/<secret>?wm_coep=on">
The app publish drawer gains a URL/Embed toggle: "URL" shows the plain
shareable link (param-free), "Embed" shows a ready-to-copy iframe
snippet with wm_coep baked in, so the flag is discoverable exactly when
embedding and absent otherwise.
`wm_coep` is consumed internally and stripped from the app `query`
context so it doesn't collide with app-defined params. Only params we
own are stripped (an explicit set), not the whole `wm_` prefix.
Fixes GIT-884
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* nit
* nit
* fix(apps): only bake wm_coep into embed snippet for raw apps
AppEditorHeaderDeploy is shared by the classic (AppEditorHeader) and raw
(RawAppEditorHeader) deploy drawers. The embed snippet unconditionally
appended ?wm_coep=on, which for a classic/low-code app forces COEP
require-corp on the document and breaks no-CORP cross-origin subresources
(external <img> in AppImage/AppStatCard/AppNavbar, {@html} embeds in
AppHtml, CDN import() in AppCustomComponent) — the exact regression the
opt-in design avoids.
Add a `rawApp` prop (default false); the raw header passes rawApp. The
flag is appended only for raw apps; classic apps get a plain iframe
snippet, and the wm_coep helper text is shown only for raw apps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
CLI deploys (sync push, set-permissioned-as) went through the same
create/update endpoints as a UI "deploy from draft", which delete the
draft at that path. That silently wiped teammates' in-progress drafts on
every push. Add a transient skip_draft_deletion deploy flag (mirroring
deployment_message) that the CLI sets; the backend then skips the
DELETE FROM draft for scripts, flows, and apps. UI deploys are unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_token_using_oauth resolved the AI OAuth resource's token_url and
POSTed to it without any SSRF validation, while base_url is validated in
get_base_url. A workspace member with resources:write could point
token_url at an internal/metadata address (e.g. 169.254.169.254),
turning the server into an authenticated blind SSRF probe.
Validate the resolved token_url with validate_url_for_ssrf before the
request, gated behind the same ALLOW_PRIVATE_AI_BASE_URLS opt-in as
base_url so private AI deployments keep working consistently for both
URL fields. ALLOW_PRIVATE_AI_BASE_URLS is now pub so windmill-api can
reuse it instead of re-parsing the env var.
* fix: trigger git sync for re-encrypted secrets on encryption key change
When changing a workspace encryption key, the secret variables get
re-encrypted with the new key, but the git sync was only dispatched for
the encryption_key.yaml metadata file. Repos with Secrets sync enabled
were left with stale ciphertexts until the next per-variable deployment.
Now, after the transaction commits, we also dispatch a Variable git sync
event for each re-encrypted secret so the new encrypted values are
pushed to the configured repos. Errors are logged but don't roll back
the key rotation.
Fixes WIN-1994
Fixes#9344
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: batch encryption-key rotation into one git-sync job
Workspace encryption key rotation now re-encrypts every secret variable
and then dispatches a single batched git-sync job carrying the Key event
plus one Variable item per re-encrypted secret. Repos with Secrets sync
enabled receive every new ciphertext in one commit instead of nothing
(previously only `encryption_key.yaml` was pushed) — and instead of N
separate jobs the debouncer might or might not merge.
Wires through the new `handle_deployment_metadata_batch` entry point
added in the companion EE PR; OSS has a no-op shim so the build stays
green.
Adds an integration test (`workspace_encryption_key_git_sync`) asserting
that rotating the key with 3 secret variables in scope produces exactly
one deployment-callback job whose `items` array contains the Key event
+ all 3 variable entries and `skip_secret=false`.
Fixes WIN-1994
Fixes#9344
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref for git-sync helper simplification
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: cover non-debouncing git-sync fallback on key rotation
Adds a regression test exercising a workspace whose sync script predates
hub version 28103: the rotation must still queue a legacy-format
deployment-callback job per item (encryption_key + each re-encrypted
secret) instead of silently skipping the repo. Bumps ee-repo-ref to the
EE fallback fix.
Addresses the P1 raised in the PR review (Codex/Pi/Claude): batch path
dropped git sync entirely for repos without sync-job debouncing support.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: add sqlx offline cache for encryption-key git-sync test queries
The cargo_test CI job builds with SQLX_OFFLINE=true; the two new
sqlx::query!/query_as! calls in
windmill-api-integration-tests/tests/workspace_encryption_key_git_sync.rs
had no cached entries, failing the build with E0282. Regenerated and
added only the two new query caches (no EE/feature cache loss).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to updated EE companion PR (08e3b9b)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ai): disable redirect following on AI proxy client to close SSRF
The AI proxy validates the configured base_url against SSRF rules but the
shared HTTP client followed up to 10 redirects without revalidating the
hops, so a public base_url could 3xx the server into a private/internal
address (e.g. the Docker socket or cloud metadata). Disable redirect
following so the validated host is the only one the server connects to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai): remove heavy redirect SSRF integration test
Drop the integration-test-level regression for redirect following; it
spins up a full API server + DB for a one-line client-config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): don't follow symlinks when reading service log files
Defense in depth on top of the existing `..` path-traversal check in
the get_log_file handler: reject the request if the final path
component is a symlink, so a planted symlink in the logs directory
cannot be used to read arbitrary files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): authorize and harden the jobs_u get_log_file endpoint
The unauthenticated jobs_u get_log_file endpoint served any job's log
file to anyone who knew the job UUID, with no authorization. Gate it the
same way as get_job_logs: look up the job (the log directory name is the
job id) filtered by workspace and the caller's scope tags, and only allow
non-logged-in callers to read logs of jobs created by the anonymous user.
Also add defense in depth: refuse to read through a symlink so a planted
symlink in the logs directory cannot be used to exfiltrate arbitrary files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fork-branch deployment callback runs the hub script
sync-script-to-git-repo-windmill, which imports `windmill-cli` as a pinned
npm dependency and runs it in-process (it does NOT shell out to a PATH wmill).
hub/28236 pinned windmill-cli@1.706.1, whose `git-deploy --only-create-branch`
path returns early without pushing — so the fork branch was checked out
locally but never published to the remote. #9366 fixed the CLI and shipped it
as windmill-cli@1.712.0, but without a hub-script bump the running callback
still used 1.706.1.
Bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28238, which is identical to 28236
except it pins windmill-cli@1.712.0 (content + lockfile). This fixes
test_workspace_fork_creates_branch and production fork-branch creation.
Also add backend/windmill-common/src/workspaces.rs to the git-sync-test
path-gate so future script-path bumps trigger the e2e (the bump alone is not
otherwise covered by the gate).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(oauth): support per-provider sandbox URLs in registry + instance settings
* fix(oauth): polish sandbox review nits (cc lookup, header label, ee ref)
* refactor(oauth): drop dead build_oauth_clients duplicate in windmill-oauth
* refactor(oauth): derive sandbox-capable provider list from registry
* chore(docker): copy oauth_connect.json into frontend build stage
* test(oauth): cover sandbox helpers (as_sandbox, canonical_name, resolve)
* chore: update ee-repo-ref to 9297d8f790346e6a6ad540c7bca1a67f91ec11a2
This commit updates the EE repository reference after PR #595 was merged in windmill-ee-private.
Previous ee-repo-ref: 3ab3eca9ac15ebab6db991e7964bc5e48ce21f42
New ee-repo-ref: 9297d8f790346e6a6ad540c7bca1a67f91ec11a2
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* [ee] fix(git-sync): publish fork branch on only_create_branch from the CLI
Fixes WIN-1997. Forking a git-sync-configured workspace must push a
`wm-fork/<branch>/<id>` branch to the repo, but the integration test
`test_workspace_fork_creates_branch` failed: the fork callback job
succeeded yet no branch appeared.
Root cause: the fork-branch callback runs the sync script with
`only_create_branch: true` and no items. The hub sync script delegates
branch checkout to `wmill sync git-deploy --only-create-branch` and runs
its own in-process commit+push ONLY for the `!only_create_branch` path
(`if (!only_create_branch) git_push(...)`). #9284 had moved commit+push
out of the CLI to the caller for the GPG-cache-warmth invariant
(WIN-1974) — but it also dropped the CLI's push for the branch-only
case. A branch-only publish has no commit, so no signing is involved and
the GPG concern does not apply; with neither the CLI nor the hub script
pushing, the empty fork branch was never published.
Restore the CLI push for the `only_create_branch` path (a bare
`git push --porcelain` of the checked-out branch ref). Adds a
deterministic CLI regression test that runs `git-deploy
--only-create-branch` for a fork workspace and asserts the branch
reaches the remote with no caller-side push.
EE companion: format the fork-branch commit message with Display instead
of Debug (no more `Some("...")` leak).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
This commit updates the EE repository reference after PR #597 was merged in windmill-ee-private.
Previous ee-repo-ref: 8b02336fcebdfae4b9d2795cbb74fa7046530bcb
New ee-repo-ref: a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit updates the EE repository reference after PR #597 was merged in windmill-ee-private.
Previous ee-repo-ref: 55c19293232be379a3044eb78f677b545882ffd6
New ee-repo-ref: a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
Automated by sync-ee-ref workflow.
* [ee] feat(queue): duration-weighted fairness admission atomic
Add the `WORKSPACE_FAIRNESS_ADMISSION_PPM` atomic that the EE
`workspace_fairness_ee::refresh_overloaded` writes on each refresh
(see companion EE PR). The atomic is read on every pull by
`should_admit_capped` to decide whether the dispatch goes down the
standard or fairness path. Defaults to 10_000 (= admit all) so the
pre-fairness behaviour is preserved until the first refresh fires.
OSS stub in `workspace_fairness.rs` continues to return `true`
unconditionally, so non-EE builds are bit-identical.
* docs(queue): consolidate full fairness algorithm into workspace_fairness.rs
Move the algorithm doc — what "overloaded" means in worker-seconds, the
duration-weighted admission derivation, coordinated refresh structure,
audit emission, the SQL perf constraints (no params CTE, drive running
side from v2_job_runtime), and EE gating — into the OSS surface module
where it is readable without EE access. The EE file becomes implementation
only.
Also bump ee-repo-ref to the EE commit that strips the duplicate doc.
* docs(queue): clarify ADMISSION_PPM default is "admit all", not count-based
Addresses CI review (claude[bot]): the `10_000` initial value is the
"admit all" no-op default that applies before the first refresh
classifies an overloaded set — not the count-based value (which would
be `target * 10_000`). The count-based form is the empty-bucket fallback
inside `compute_admission_ppm`, a different thing.
* chore(queue): point ee-repo-ref at EE main (fairness admission merged via #593)
* fix(queue): duration-weighted admission uses unclamped service-time window
Bumps ee-repo-ref to the EE fix (windmill-ee-private#596) that sources
D_c/D_u for the admission probability from a separate 60s service-time
window of true `duration_ms`, instead of the occupancy aggregation whose
per-job contributions are clamped to the 10s occupancy window. The clamp
truncated D_c for capped jobs longer than the window, under-admitting the
duration skew (true 34s jobs → ~86% effective share instead of the target
65%). Occupancy worker-seconds still drive overload classification.
Updates the algorithm doc in workspace_fairness.rs accordingly.
Note: ee-repo-ref points at the EE feature branch; re-point to EE main
once #596 merges.
`announce_server_started` writes a `server_heartbeat:{INSTANCE_NAME}` row
on each startup. INSTANCE_NAME is a fresh random string per process, so
the row is never updated again and a new row is inserted on every
restart, growing background_task_state unboundedly.
Add an hourly monitor task that deletes server_heartbeat:* rows older
than 7 days. Older rows cannot influence check_any_server_started (which
only considers heartbeats refreshed after the restart was initiated), so
they are safe to prune.
Fixes WIN-1990.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(queue): bump EE ref to include worker_ping fairness signal
The current ee-repo-ref.txt pointed to 31cda7c (an unrelated merge
commit on the asset-graph-view-ee branch) instead of ddc9e80, which
contains the workspace-fairness fix that switches the active-share
signal from v2_job_queue.running=true to worker_ping. As a result
cloud was still computing overload off the legacy signal, so a
workspace with many in-flight/suspended flows (lancom01-prod, with
799 suspended flows × 3 v2_job_queue bookkeeping rows each = 2397
running-true rows) was flagged as 95% of cluster activity despite
consuming zero worker slots.
Bumping to ddc9e80 picks up the worker_ping-based signal, which
naturally excludes (a) suspended jobs (no worker pinging them),
(b) zombie running-rows from dead workers, and (c) flow/flownode
orchestration rows that never run on a worker in the first place.
* test(queue): seed v2_job rows + realistic durations for fairness helpers
The new duration-weighted fairness algorithm joins v2_job_queue and
v2_job_completed to v2_job for the `kind` filter (excluding flow
bookkeeping) and reads `duration_ms` for the completed contribution.
Update the test helpers to mirror that schema:
* `insert_completed` now inserts a matching v2_job row (kind=script)
and writes `duration_ms = 1000` with a 1-second [started_at,
completed_at] interval, so each completed row contributes ~1
worker-second when fully inside the refresh window.
* `insert_queued` likewise pre-inserts v2_job, sets `started_at`
to NOW() - 1s when running=true (so running rows contribute ~1
worker-second by the time the refresh runs), and seeds
v2_job_runtime.ping so the running side accrues real-time worker
seconds (the algorithm bounds end-of-interval by ping).
The zombie/suspended insert helpers are intentionally left without
v2_job rows — the new algorithm's INNER JOIN excludes them, so they
still correctly contribute zero worker-seconds.
* chore(queue): bump EE ref to duration-weighted fairness algorithm
Companion to windmill-ee-private#<TBD>: switch the EE workspace
fairness aggregation from a count-based UNION (worker_ping snapshot
+ v2_job_completed count) to a worker-seconds aggregation sourced
directly from v2_job_queue and v2_job_completed, with kind/suspend
filters mirroring handle_zombie_jobs and per-row defenses against
zombie inflation on both halves.
* chore(queue): bump EE ref for fairness perf fix (inline window_start)
* chore(queue): bump EE ref for fairness perf rewrite (driver-side flip)
* update ee ref
`fairness_ignores_zombie_running_rows` and
`fairness_ignores_concurrency_suspended_rows` panic intermittently in CI
(both Linux and Windows runs). Mark them `#[ignore]` until the
underlying flakiness is resolved.