windmill/docs/perf-profiling.md

# Profiling the per-job path

How to find and prove a performance improvement on the job path, which is what any run of a
script or flow pays: push, pull, execution, log writes, completion. The method finds candidates
by measuring rather than by reading code, and keeps only changes whose effect was measured
before and after. A candidate that cannot be measured is not a finding.

## The harness

`backend/tests/perf_profile.rs` runs real jobs through the real queue, worker and database, all
in the test process. Its tests are `#[ignore]`d, so CI only compiles them. For each workload it
reports the following, separately for the idle, push and worker phases:

- requested allocation bytes and allocation count per job (raw, and with the idle rate subtracted)
- allocation sites, sampled once every 128 KiB allocated and once every 2048 allocations. Each
  site is shown as the deepest Windmill `file:line` plus the library call it allocated in
  (`LEAF`), and as the call chain of the four deepest Windmill frames (`CHAIN`)
- with `PERF_SQL=1`, instead of allocations: every SQL statement, how many times it runs per job,
  and its database time per job

Workloads: a deployed bash script, a deployed Bun script, a deployed 3-step identity flow and a
deployed 3-step bash flow. To profile another path, add a workload with the same shape: create
the script or flow with SQL, then call `profile_workload`. Typical additions are a Python
script, a script with many args, a flow with a branch or a loop, or a job with a large result.

```bash
cd backend && source ../.env.local          # DATABASE_URL: sqlx::test creates its DBs from it
OUT=/tmp/perf && mkdir -p $OUT
# allocations and sites
PERF_OUT=$OUT PERF_N=30 cargo test --features quickjs --test perf_profile -- --ignored --nocapture --test-threads=1
# SQL statements per job (a separate run, see the caveats below)
PERF_SQL=1 PERF_OUT=$OUT/sql PERF_N=30 cargo test --features quickjs --test perf_profile -- --ignored --nocapture --test-threads=1
grep -h '^===' $OUT/*.txt                   # the per-job totals, one line per phase
```

Use the same `--features` as the running dev backend so the build reuses its artifacts. One
workload takes about 20 s.

## Reading the output

- **Bytes per job is the main metric.** It is very repeatable, varying less than 0.1% between
  runs on the same code, so 3 runs before and 3 after a change are enough. Time per job is noisy
  in a debug build and is not a measure of anything.
- **A site that sits on top of every script workload deserves attention first.** Per-job setup
  that costs the same however small the job is matters most, because most jobs are small.
- **Discount `pin<…>` / `Box::pin` sites.** They are async state machines, which are many times
  larger without optimisation. Confirm one in release before acting on it.
- **SQL statements per job is usually the larger lever.** Each statement is a network round trip
  in production. Look for statements repeated per job or per step that read something already
  known (a cached value, a row read earlier in the same job), and for appends that could be
  merged.

## A/B a candidate

1. Run the relevant workload 3 times on `main` and keep the `WORKER` (or `PUSH`) lines.
2. Apply the change, then run it 3 times again.
3. When the cost is CPU in a library, also confirm it in a release build: a small standalone
   crate in `/tmp` that pins the same crate version (see `Cargo.lock`), with a counting global
   allocator and a timing loop. Debug numbers measure allocations reliably, but they do not
   measure CPU.
4. Run the tests that pin the behaviour you touched, and put the before/after table in the PR.

Changes to job completion, the queue pull or the flow state machine are the critical path. Test
them beyond the profiler: the existing suites for that area, concurrency, and older Postgres
versions.

## Caveats

- **Split DWARF.** In dev builds a frame's location is a relative `./windmill-*/src/…:line`, and
  inlined frames have bare names (`poll<…>`). The harness keys sites by location for that reason.
  Do not switch it to keying by symbol name.
- **SQL counting changes the allocation numbers.** Once `sqlx::query` events are enabled, sqlx
  formats a summary for every statement. Never read allocations from a `PERF_SQL=1` run.
- **The sampler's own work is excluded.** `IN_SAMPLER` stops the backtrace's allocations from
  being counted. Keep it if you change how samples are captured, or idle reads tens of MB/s.
- **`RUST_BACKTRACE` and `RUST_LIB_BACKTRACE` must be unset (or `0`).** Sampling takes std's
  backtrace lock, which is not reentrant. With either variable enabled, errors capture
  backtraces of their own, and an allocation made under that lock can deadlock the sampler.
  The harness refuses to start when either is set.
- **Deployed Bun scripts need a NULL lock.** `''` fails at run time with "expected //bun.lock
  split pattern".
- **Disk.** The test build shares `target/` with the dev backend, but the first build also
  compiles the test-only dependencies. Check `df -h /` first.