mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-03 08:02:19 +00:00
docs: make herdr the documented worktree manager (#10727)
* chore: share worktree port assignment across worktree tooling The slot scan and the .env.local writer existed twice, once inline in scripts/worktree-env and once in scripts/post-create.sh. Both are now wm_assign_ports and wm_write_env_local in scripts/worktree-common.sh, so a third caller cannot drift from the other two. wm_assign_ports takes the worktree root instead of reading the process cwd, which lets it run from outside the worktree, and prints its slot line to stderr so stdout carries only the port pair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: describe both worktree managers in the dev environment section The section assumed webmux and tmux throughout. Worktree tooling is being trialled on herdr alongside it, so the detection markers and the pane inspection commands now appear side by side. Corrects the pointer for per-worktree values: .env.local is written by both managers through wm_write_env_local and is readable, so it is the portable source. webmux's runtime.env stays documented as the place carrying the extras .env.local lacks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: point at the herdr plugin setup Naming the repo left the reader to work out what to do with it. The plugins do nothing until they are installed and their actions are bound, and Herdr has no action palette to discover them through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: narrow the shared helper to .env.local writing Slot assignment goes back into worktree-env: it had one caller, and the herdr plugin it was extracted for provisions worktrees before the checkout is usable, so it carries its own copy rather than sourcing one whose contents depend on whichever branch the main checkout sits on. Writing .env.local stays shared - worktree-env and post-create.sh each had the same eleven lines. Also corrects AGENTS.md: .env.local is a shared format rather than a shared function, and WM_DB_NAME comes from it rather than webmux's runtime.env. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: correct the .env.local readability claim The file-read tool denies every .env.* path via .claude/settings.json, .env.local included, so the section points at cat instead of implying the file is readable. DATABASE_URL carries an export prefix, which the bare 'source .env.local' in .envrc needs to reach cargo and sqlx, and which a ^DATABASE_URL= grep would miss. Documented rather than removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: target the backend pane in the webmux capture-pane cell Without -t the command dumps the agent's own pane. Matches the form .webmux.yaml already uses in its startup prompt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: make herdr the documented worktree manager Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: split how the two managers pick up a CARGO_FEATURES change Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
56a6ad322e
commit
4f7f705f8c
@@ -210,7 +210,7 @@ the diff first — `git diff --name-only main...HEAD` answers most of these.
|
||||
**Flip without asking** when it is self-contained: a single-file fix, test-only, docs-only, one
|
||||
call site, no new public surface.
|
||||
|
||||
Unattended (webmux oneshot) there is nobody to ask, so the judgement holds and the action
|
||||
Unattended (a one-shot run) there is nobody to ask, so the judgement holds and the action
|
||||
degrades: flip the self-contained ones, and leave the rest at a clean draft with a line in the PR
|
||||
description saying why — `left in draft: adds a migration, wants a human look before ready`.
|
||||
Don't flip a wide-blast-radius change just because the round came back clean, and don't ask a
|
||||
|
||||
@@ -55,12 +55,23 @@ Open-source platform for internal tools, workflows, API integrations, background
|
||||
> defaults in this section apply only to a plain single checkout. **Discover the real
|
||||
> values before running anything** — see "Per-worktree ports and database" below.
|
||||
|
||||
**Check whether they are already running before starting anything.** In a webmux worktree
|
||||
(`$WEBMUX_WORKTREE_PATH` is set) the backend and frontend are already up in sibling tmux panes —
|
||||
use those, don't spawn your own. `tmux list-panes -t "$(tmux display-message -p -t "$TMUX_PANE"
|
||||
'#{window_id}')" -F '#{pane_index} #{pane_current_command}'` shows what is running; read its log
|
||||
with `tmux capture-pane`, and see `backend/AGENTS.md` to restart it with different cargo features.
|
||||
A second server started in your own shell fights the first one for the port. The commands below
|
||||
**Check whether they are already running before starting anything.** In a managed worktree the
|
||||
backend and frontend are already up in sibling panes — use those, don't spawn your own. A second
|
||||
server started in your own shell fights the first one for the port.
|
||||
|
||||
**herdr** is the worktree manager; `$HERDR_ENV=1` marks one of its panes. `herdr pane list
|
||||
--workspace "$HERDR_WORKSPACE_ID"` lists this worktree's panes — match on `cwd`, since the servers
|
||||
run from `backend/` and `frontend/` — and `herdr pane read <pane_id> --source recent-unwrapped
|
||||
--lines 50` reads one's log; `herdr --skill` prints the full reference for
|
||||
inspecting and driving panes and agents. The plugins that provision these worktrees live in the
|
||||
private `windmill-labs/windmill-herdr` — clone it and run `./setup.sh` to install them and the
|
||||
keybindings they need.
|
||||
|
||||
A worktree from the older **webmux** setup sets `$WEBMUX_WORKTREE_PATH` instead and is driven
|
||||
through tmux: `tmux list-panes -t "$(tmux display-message -p -t "$TMUX_PANE" '#{window_id}')" -F
|
||||
'#{pane_index} #{pane_current_command}'` for what is running, `tmux capture-pane` for a pane's log.
|
||||
|
||||
See `backend/AGENTS.md` to restart the backend with different cargo features. The commands below
|
||||
are for a plain checkout with nothing running.
|
||||
|
||||
- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
|
||||
@@ -72,10 +83,16 @@ are for a plain checkout with nothing running.
|
||||
|
||||
### Per-worktree ports and database
|
||||
|
||||
In a webmux worktree the authoritative values live in
|
||||
`$(git rev-parse --git-dir)/webmux/runtime.env` — `BACKEND_PORT`, `FRONTEND_PORT`,
|
||||
`DATABASE_URL`, `CARGO_FEATURES`, `WM_DB_NAME`. Every pane sources it at startup. Read that
|
||||
first: it is not a `.env*` file, so the repo's secret-file read rules don't stand in the way.
|
||||
**`.env.local` in the worktree root holds the real values** — `BACKEND_PORT`, `FRONTEND_PORT`,
|
||||
`REMOTE`, `DATABASE_URL`, `CARGO_FEATURES`, `WM_DB_NAME`. Every manager writes those fields, so it
|
||||
is correct whichever one you are in. Read it with `cat .env.local` from a shell: the file-read
|
||||
tool denies every `.env.*` path, and `DATABASE_URL` is written with an `export` prefix that a
|
||||
`^DATABASE_URL=` grep misses.
|
||||
|
||||
herdr keeps nothing besides that file; its hooks write it and read it back. A webmux worktree
|
||||
additionally has `$(git rev-parse --git-dir)/webmux/runtime.env`, which every pane sources at
|
||||
startup and which carries the extras `.env.local` lacks: `WEBMUX_*`, `WM_CLONE_DB`,
|
||||
`USE_RUST_PLUGIN`.
|
||||
|
||||
In a plain checkout, fall back to `.env` / `.env.local` (repo root) and `backend/.env`.
|
||||
|
||||
@@ -84,19 +101,25 @@ hook. It is not a copy of the main dev instance: you get the `admins` workspace,
|
||||
`admin@windmill.dev` superadmin, the license key copied from the base database, and whatever the
|
||||
migrations seed — and none of your own workspaces, scripts, flows or apps. Create whatever a test
|
||||
needs. Cloning the base `windmill` database instead is
|
||||
opt-in per project via `WM_CLONE_DB` in `.webmux.yaml`; read the note there before turning it on.
|
||||
opt-in via `WM_CLONE_DB` (the `windmill.worktree` plugin's `config.env` under herdr, or
|
||||
`.webmux.yaml` under webmux); read the note in `.webmux.yaml` before turning it on.
|
||||
|
||||
The database is named after the **worktree directory, not the branch** (`scripts/worktree-common.sh`):
|
||||
`windmill_` + the directory basename with `-` → `_`, which Postgres then truncates at 63
|
||||
characters. Branch `hugo/win-2340-ai-agent-evals-standalone-agent-runs-and-eval-datasets` sits in
|
||||
a worktree directory named `win-2340-…`, so its database is
|
||||
`windmill_win_2340_ai_agent_evals_standalone_agent_runs_and_eval` — no `hugo_`, and the tail
|
||||
chopped. Take `WM_DB_NAME` from `runtime.env` instead of reconstructing the name. Read those, or
|
||||
discover from what is already running:
|
||||
characters. herdr creates the directory under `~/.herdr/worktrees/<repo>/` from the branch name
|
||||
with each `/` turned into `-`, so branch
|
||||
`hugo/win-2544-add-options-field-to-the-postgresql-resource-type` gets the database
|
||||
`windmill_hugo_win_2544_add_options_field_to_the_postgresql_reso` — the whole branch name, cut
|
||||
mid-word at the limit. The two drift apart as soon as the branch is renamed, and webmux named its
|
||||
directories after the ticket alone, so reconstructing the name from the branch you are on is
|
||||
wrong in both. Take `WM_DB_NAME` from `.env.local` instead. Read that, or discover from what is
|
||||
already running:
|
||||
|
||||
```bash
|
||||
# match on the worktree directory, truncated the way Postgres truncates it (63 - len('windmill_')):
|
||||
psql postgres://postgres:changeme@localhost:5432/postgres -tAc \
|
||||
"select datname from pg_database where datname like 'windmill%'" | grep "$(git branch --show-current | tr - _)"
|
||||
"select datname from pg_database where datname like 'windmill%'" \
|
||||
| grep -F "$(basename "$(git rev-parse --show-toplevel)" | tr '/-' '_' | cut -c1-54)"
|
||||
# the port the frontend actually proxies to (REMOTE of this worktree's vite):
|
||||
for p in $(pgrep -f vite); do case "$(readlink /proc/$p/cwd)" in *"$(basename "$(git rev-parse --show-toplevel)")"*)
|
||||
tr '\0' '\n' < /proc/$p/environ | grep -E '^REMOTE=|^PORT=';; esac; done
|
||||
|
||||
+28
-17
@@ -23,7 +23,7 @@
|
||||
## Cargo features & running the dev backend
|
||||
|
||||
The dev backend runs under `cargo watch` and is launched by default with **only
|
||||
`--features quickjs`** (see the tmux backend pane). That baseline compiles fast but
|
||||
`--features quickjs`** (see the backend pane). That baseline compiles fast but
|
||||
**deliberately omits most functionality** — notably S3/object storage, the S3 proxy, all
|
||||
EE code, MCP, and every non-JS language runtime. A running server never gains a feature you
|
||||
didn't compile in: feature-gated routes 404 or return a `"requires <feature>"` stub. So if
|
||||
@@ -33,22 +33,30 @@ MUST **restart the backend with the appropriate features** for what you're worki
|
||||
### Restarting the dev backend with the right features
|
||||
|
||||
Restart in the **same pane**, so the relaunch inherits that pane's `DATABASE_URL`, `BACKEND_PORT`
|
||||
and the rest of `runtime.env`. Scope every kill to this worktree — **never**
|
||||
and the rest of the worktree environment. Scope every kill to this worktree — **never**
|
||||
`pkill -f target/debug/windmill`, which kills every sibling worktree's backend.
|
||||
|
||||
1. Find the backend pane by what it is running, not by index. The index depends on the webmux
|
||||
profile: pane 1 is the backend under `full`, but the *frontend* under `frontendOnly`.
|
||||
1. Find the backend pane by where it is running, not by index. Under herdr the two server panes
|
||||
sit in the worktree's `backend/` and `frontend/`, so match on `cwd`:
|
||||
|
||||
```bash
|
||||
herdr pane list --workspace "$HERDR_WORKSPACE_ID"
|
||||
```
|
||||
|
||||
Under webmux they are tmux panes, and the index depends on the profile: pane 1 is the backend
|
||||
under `full`, but the *frontend* under `frontendOnly`.
|
||||
|
||||
```bash
|
||||
WIN=$(tmux display-message -p -t "$TMUX_PANE" '#{window_id}')
|
||||
tmux list-panes -t "$WIN" -F '#{pane_index} #{pane_current_command} #{pane_pid}'
|
||||
```
|
||||
|
||||
2. Read the feature set it is **actually** running. `CARGO_FEATURES` in `runtime.env` is only what
|
||||
the pane started with, and goes stale the first time anyone restarts by hand:
|
||||
2. Read the feature set it is **actually** running. The `CARGO_FEATURES` the worktree was created
|
||||
with is only what the pane started with, and goes stale the first time anyone restarts by hand:
|
||||
|
||||
```bash
|
||||
ps --ppid <pane_pid> -o args=
|
||||
herdr pane process-info --pane <pane_id> # foreground_processes[].argv
|
||||
ps --ppid <pane_pid> -o args= # under webmux
|
||||
# /home/hugo/.cargo/bin/cargo-watch watch -x run --features quickjs
|
||||
```
|
||||
|
||||
@@ -56,22 +64,25 @@ and the rest of `runtime.env`. Scope every kill to this worktree — **never**
|
||||
explicitly:
|
||||
|
||||
```bash
|
||||
tmux send-keys -t "$WIN.<idx>" C-c
|
||||
tmux send-keys -t "$WIN.<idx>" 'PORT=$BACKEND_PORT cargo watch -x "run --features quickjs,private,parquet"' Enter
|
||||
herdr pane send-keys <pane_id> C-c
|
||||
herdr pane run <pane_id> 'PORT=$BACKEND_PORT cargo watch -x "run --features quickjs,private,parquet"'
|
||||
# under webmux: tmux send-keys -t "$WIN.<idx>" C-c, then the same command line with Enter
|
||||
```
|
||||
|
||||
Carry over every feature the old command had unless you mean to drop one — rebuilding the list
|
||||
from memory is how a backend silently loses `quickjs`.
|
||||
|
||||
4. Persist the new set so a recreated pane starts with it: set `CARGO_FEATURES` in the
|
||||
worktree's `.env.local`, which `scripts/post-create.sh` writes and webmux reads. Do **not**
|
||||
edit `runtime.env` for this — webmux regenerates it from metadata and `.env.local` every time
|
||||
the worktree is opened, so an edit there is lost on the next reopen. Either way the change
|
||||
only affects a future pane; step 3 is what takes effect now.
|
||||
4. Persist the new set so a recreated pane starts with it: set `CARGO_FEATURES` in the worktree's
|
||||
`.env.local`, which the worktree hooks write. A herdr pane sources that file directly, so the
|
||||
next pane picks the value up. A webmux pane sources `webmux/runtime.env` instead, which webmux
|
||||
regenerates from metadata and `.env.local` when the worktree is *opened* — so the value lands
|
||||
on the next reopen, not the next pane, and editing `runtime.env` by hand is lost at that same
|
||||
reopen. Either way the change only affects a future pane; step 3 is what takes effect now.
|
||||
|
||||
5. Re-capture the pane until `health check completed` appears before hitting the API. A cold
|
||||
rebuild takes ~60s, and the previous run's success line is still in the scrollback, so a
|
||||
capture taken too early reads as ready when it isn't.
|
||||
5. Re-read the pane (`herdr pane read <pane_id> --source recent-unwrapped`, or `tmux
|
||||
capture-pane`) until `health check completed` appears before hitting the API. A cold rebuild
|
||||
takes ~60s, and the previous run's success line is still in the scrollback, so a capture taken
|
||||
too early reads as ready when it isn't.
|
||||
|
||||
cargo-watch only re-runs on a file change, so after an idle/failed run `touch README.md` (from
|
||||
`backend/`, where the watch runs) is a cheap retrigger (touching a `.rs` forces a full rebuild).
|
||||
|
||||
Reference in New Issue
Block a user