starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-21 16:02:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(headless-server): add upgrade SOP for orca serve on Linux (#9575)

Browse Source 
* docs(headless-server): add upgrade SOP for orca serve on Linux

The headless Linux guide covered install/run/systemd but had no upgrade
section, leaving operators to guess how to move to a new AppImage without
losing state.

Add an "Upgrade" section documenting the manual SOP (serve mode never
auto-updates) and one troubleshooting bullet:

- State lives under the service user's ~/.config (orca + Orca dirs),
  independent of /opt/orca, and orca-data.json is forward-migrated on load,
  so a forward upgrade is safe.
- Replace the binary with an atomic same-filesystem rename (download to
  .new, verify, mv) — never curl -o over the FUSE-mounted live binary.
- Back up the whole .config before upgrading, because rollback is NOT
  binary-only safe: an older build strips newer orca-data.json fields it
  doesn't recognize, and the .bak.* ring is corruption-recovery, not a
  pre-upgrade copy.
- Note there is no headless version command; track the release tag instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(headless-server): harden the orca serve upgrade/rollback runbook

Address CodeRabbit review on #9575:

- Fail closed: run the upgrade block under `set -euo pipefail`, remove any stale
  `.new` file before download, and gate the atomic `mv` on an explicit ELF check
  so a failed/partial/non-ELF download can never be promoted.
- Keep /opt/orca/VERSION tied to the installed binary: a single `TAG` variable
  drives both the download URL and the recorded VERSION, saved as VERSION.prev on
  upgrade and restored on rollback so the audit file never drifts.
- Crash-loop troubleshooting now points to Roll back first (restores the
  pre-upgrade orca-data.json) instead of re-running Upgrade.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(headless): harden server upgrade SOP

---------

Co-authored-by: fanyunqian.1 <fanyunqian.1@bytedance.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
This commit is contained in:
Yunqian Fan
2026-07-20 21:38:39 -04:00
committed by GitHub
co-authored by Claude Opus 4.8 fanyunqian.1 OrcaWin
parent 6444be3a01
commit aad34cbb32
1 changed files with 439 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/headless-linux-server.md
+439 -3
View File
@@ -15,7 +15,7 @@ Install the AppImage runtime dependency and Xvfb:
```bash
sudo apt-get update
sudo apt-get install -y curl libfuse2 xvfb
sudo apt-get install -y curl file libfuse2 xvfb
```
Download and make the AppImage executable:
@@ -56,11 +56,14 @@ The command prints the runtime endpoint and pairing URL. Stop it with `Ctrl+C`.
## Systemd Service
Create a dedicated service user and install directory. Run the service as this
user instead of root so the AppImage can keep Chromium's sandbox enabled.
user instead of root so the AppImage can keep Chromium's sandbox enabled. Keep
the install directory root-owned: the service needs to read and execute the
AppImage, but must not be able to replace it or the rollback artifacts.
```bash
sudo useradd --system --create-home --shell /usr/sbin/nologin orca
sudo chown -R orca:orca /opt/orca
sudo chown root:root /opt/orca /opt/orca/orca-linux.AppImage
sudo chmod 755 /opt/orca /opt/orca/orca-linux.AppImage
```
For most hosts, one `orca serve` service is enough because Orca starts Xvfb on
@@ -165,6 +168,436 @@ If you later install the desktop CLI from Orca settings, use that CLI for normal
shell workflows. Keep the AppImage path in systemd so service restarts do not
depend on an interactive shell profile.
## Upgrade
`orca serve` never updates itself. In headless mode Orca wires up no auto-updater
at all — the built-in updater only runs in the desktop GUI, and no paired mobile
or web client can trigger it remotely. Upgrading is always a deliberate step:
replace the AppImage and restart the service.
Two facts make this safe and predictable:
- **State lives in the service user's home, not next to the binary.** Persisted
data is under `/home/orca/.config/` (Orca uses both an `orca` and an `Orca`
directory there), fully independent of `/opt/orca/orca-linux.AppImage`.
Replacing the binary never touches projects, worktree metadata, terminal
history, orchestration state, or paired-device keys — so mobile and web
clients reconnect after an upgrade without re-pairing.
- **New builds migrate old state on load.** Orca loads older `orca-data.json`
state into the current schema and writes it back in the current shape, so a
forward upgrade needs no manual data step.
Rolling back is the case that needs care — see [Roll back](#roll-back).
### Record the version you deploy
Orca has no headless version command: there is no `--version` flag or `version`
subcommand, and `orca serve` prints only its endpoint. Choose a release tag
explicitly instead of following the `latest` URL, and record it next to the
binary so upgrades are auditable. The steps below keep that record in
`/opt/orca/VERSION`.
### Upgrade steps
Never download straight onto `/opt/orca/orca-linux.AppImage`. The AppImage is
FUSE-mounted, so overwriting it in place while the service runs can crash or
corrupt the live process — and even with the service stopped, a failed or partial
download would clobber the working binary. Instead download to a temporary name
on the same filesystem, verify it, then swap it in with an atomic rename.
Check capacity before starting:
```bash
sudo chown root:root /opt/orca
sudo chmod 755 /opt/orca
sudo test ! -L /opt/orca/orca-linux.AppImage
sudo chown root:root /opt/orca/orca-linux.AppImage
sudo chmod 755 /opt/orca/orca-linux.AppImage
# Clear predictable staging names left by an older attempt after locking the directory
sudo rm -f /opt/orca/orca-linux.AppImage.new /opt/orca/VERSION.new \
/opt/orca/orca-linux.AppImage.recovering /opt/orca/VERSION.recovering
sudo du -sh /home/orca/.config
df -h /opt/orca /home/orca
```
`/opt/orca` needs room for the compressed Orca profile archive, the staged
build, and the rollback binary. A rollback extracts the old profile and preserves
the post-upgrade Orca profile directories, so `/home` needs room for both copies.
Run the following block as one Bash script so its fail-fast and recovery traps
remain active for the whole operation:
```bash
set -euo pipefail
# Replace this example with the release tag you intend to deploy
ORCA_VERSION=v1.4.147
# Select the release asset on the server where Orca runs
case "$(uname -m)" in
x86_64)
ORCA_ASSET=orca-linux.AppImage
ORCA_FILE_MACHINE=x86-64
;;
aarch64 | arm64)
ORCA_ASSET=orca-linux-arm64.AppImage
ORCA_FILE_MACHINE='ARM aarch64'
;;
*)
echo "Unsupported architecture: $(uname -m)" >&2
exit 1
;;
esac
ORCA_ROLLBACK_NEW=
ORCA_ROLLBACK=
ORCA_SERVICE_STOPPED=0
ORCA_BINARY_PROMOTED=0
recover_failed_upgrade() {
exit_status=$?
trap - EXIT
set +e
if ((exit_status != 0)); then
sudo rm -f /opt/orca/orca-linux.AppImage.new /opt/orca/VERSION.new \
/opt/orca/orca-linux.AppImage.recovering /opt/orca/VERSION.recovering
fi
if ((exit_status != 0)) && [[ -n "$ORCA_ROLLBACK_NEW" ]] && \
sudo test -d "$ORCA_ROLLBACK_NEW"; then
sudo rm -rf -- "$ORCA_ROLLBACK_NEW"
fi
if ((exit_status != 0 && ORCA_SERVICE_STOPPED)); then
recovery_ok=1
if ((ORCA_BINARY_PROMOTED)); then
if ! sudo cp -a "$ORCA_ROLLBACK/orca-linux.AppImage" \
/opt/orca/orca-linux.AppImage.recovering || \
! sudo mv -f /opt/orca/orca-linux.AppImage.recovering \
/opt/orca/orca-linux.AppImage; then
recovery_ok=0
fi
if sudo test -f "$ORCA_ROLLBACK/VERSION"; then
if ! sudo cp -a "$ORCA_ROLLBACK/VERSION" /opt/orca/VERSION.recovering || \
! sudo mv -f /opt/orca/VERSION.recovering /opt/orca/VERSION; then
recovery_ok=0
fi
elif ! sudo rm -f /opt/orca/VERSION; then
recovery_ok=0
fi
fi
sudo rm -f /opt/orca/orca-linux.AppImage.recovering \
/opt/orca/VERSION.recovering
if ((recovery_ok)); then
sudo systemctl start orca-serve.service || true
else
echo 'Upgrade recovery failed; service remains stopped' >&2
fi
fi
exit "$exit_status"
}
trap recover_failed_upgrade EXIT
# 1. Stage and verify the new build while the server stays online
sudo curl -fL --retry 3 "https://github.com/stablyai/orca/releases/download/${ORCA_VERSION}/${ORCA_ASSET}" \
-o /opt/orca/orca-linux.AppImage.new
sudo chown root:root /opt/orca/orca-linux.AppImage.new
sudo chmod 755 /opt/orca/orca-linux.AppImage.new
# Both checks must match; either grep stops this fail-fast block otherwise
ORCA_FILE_INFO=$(LC_ALL=C file /opt/orca/orca-linux.AppImage.new)
grep 'ELF .* executable' <<<"$ORCA_FILE_INFO"
grep -F "$ORCA_FILE_MACHINE" <<<"$ORCA_FILE_INFO"
# 2. Assemble the prior binary and version in a root-only rollback bundle
ORCA_ROLLBACK_BASE=/opt/orca/orca-rollback-$(date +%F-%H%M%S-%N)
ORCA_ROLLBACK_NEW=${ORCA_ROLLBACK_BASE}.new
ORCA_ROLLBACK=${ORCA_ROLLBACK_BASE}.ready
sudo install -d -m 700 "$ORCA_ROLLBACK_NEW"
sudo cp -a /opt/orca/orca-linux.AppImage "$ORCA_ROLLBACK_NEW/orca-linux.AppImage"
if sudo test -f /opt/orca/VERSION; then
sudo cp -a /opt/orca/VERSION "$ORCA_ROLLBACK_NEW/VERSION"
fi
# Stage the new version record before the stop window
printf '%s\n' "$ORCA_VERSION" | sudo tee /opt/orca/VERSION.new >/dev/null
sudo chown root:root /opt/orca/VERSION.new
sudo chmod 644 /opt/orca/VERSION.new
# 3. Stop the server so the profile backup is consistent
ORCA_SERVICE_STOPPED=1
sudo systemctl stop orca-serve.service
# Add only Orca-owned profile directories, then publish the complete bundle
ORCA_PROFILE_DIRS=()
for profile_dir in orca Orca; do
if sudo test -L "/home/orca/.config/$profile_dir"; then
echo "Refusing symlinked Orca profile: /home/orca/.config/$profile_dir" >&2
exit 1
fi
if sudo test -d "/home/orca/.config/$profile_dir"; then
if [[ "$profile_dir" == Orca ]] && \
sudo test /home/orca/.config/orca -ef /home/orca/.config/Orca; then
continue
fi
ORCA_PROFILE_DIRS+=("$profile_dir")
fi
done
if ((${#ORCA_PROFILE_DIRS[@]} == 0)); then
echo 'No Orca profile directory found under /home/orca/.config' >&2
exit 1
fi
sudo tar czf "$ORCA_ROLLBACK_NEW/profile.tgz" \
-C /home/orca/.config "${ORCA_PROFILE_DIRS[@]}"
sudo chmod 600 "$ORCA_ROLLBACK_NEW/profile.tgz"
sudo mv "$ORCA_ROLLBACK_NEW" "$ORCA_ROLLBACK"
# 4. Atomically replace the binary and version record, then start
ORCA_BINARY_PROMOTED=1
sudo mv -f /opt/orca/orca-linux.AppImage.new /opt/orca/orca-linux.AppImage
sudo mv -f /opt/orca/VERSION.new /opt/orca/VERSION
sudo systemctl start orca-serve.service
ORCA_SERVICE_STOPPED=0
trap - EXIT
```
The profile archive created in step 3 captures both Orca profile directory names
when present without rewinding unrelated tools under `/home/orca/.config`. The
`.ready` suffix is published only after the prior binary, version record, and
profile archive are complete. If you run the managed Xvfb unit, only
`orca-serve.service` needs restarting — leave `orca-xvfb.service` running.
### Verify
```bash
sudo journalctl -u orca-serve.service -f
```
A healthy start prints `Orca server ready: ws://0.0.0.0:6768` (with your port).
Confirm a client reconnects before you discard the backup. The timestamped
rollback bundles are not pruned automatically. After the new version satisfies
your retention policy, select and inspect the newest complete bundle before
removing it:
```bash
shopt -s nullglob
ORCA_ROLLBACK_SETS=(/opt/orca/orca-rollback-*.ready)
((${#ORCA_ROLLBACK_SETS[@]} > 0))
ORCA_ROLLBACK=${ORCA_ROLLBACK_SETS[${#ORCA_ROLLBACK_SETS[@]} - 1]}
printf 'Removing rollback bundle: %s\n' "$ORCA_ROLLBACK"
sudo test -d "$ORCA_ROLLBACK"
sudo rm -rf -- "$ORCA_ROLLBACK"
```
Each `.ready` directory is a self-contained rollback generation; never combine
files from different bundles.
### Roll back
A rollback is **not** binary-only safe. Once a newer build has started, it can
rewrite `orca-data.json` in the current schema. If an older build then writes
that file, it can discard fields it does not recognize. The rolling
`orca-data.json.bak.*` files are corruption-recovery snapshots, not a dedicated
pre-upgrade copy, and normal writes can rotate them away. To roll back cleanly,
restore the backup from step 3 **and** swap the binary back. Run this block as one
Bash script:
```bash
set -euo pipefail
# Select and validate one complete generation before taking the service offline
shopt -s nullglob
ORCA_ROLLBACK_SETS=(/opt/orca/orca-rollback-*.ready)
((${#ORCA_ROLLBACK_SETS[@]} > 0))
ORCA_ROLLBACK=${ORCA_ROLLBACK_SETS[${#ORCA_ROLLBACK_SETS[@]} - 1]}
sudo test -f "$ORCA_ROLLBACK/orca-linux.AppImage"
sudo tar tzf "$ORCA_ROLLBACK/profile.tgz" >/dev/null
# Extract and validate the old profile while the current server stays online
sudo test ! -L /home
ORCA_HOME_OWNER=$(sudo stat -c %u /home)
ORCA_HOME_MODE=$(sudo stat -c %a /home)
if [[ "$ORCA_HOME_OWNER" != 0 ]] || ((8#$ORCA_HOME_MODE & 0022)) || \
sudo -u orca test -w /home; then
echo 'Refusing rollback because /home is not root-controlled' >&2
exit 1
fi
ORCA_RESTORE=$(sudo mktemp -d /home/.orca-restore.XXXXXX)
ORCA_SERVICE_STOPPED=0
ORCA_MOVED_CURRENT_DIRS=()
ORCA_INSTALLED_RESTORE_DIRS=()
ORCA_CURRENT_BINARY_MOVED=0
ORCA_CURRENT_VERSION_MOVED=0
ORCA_VERSION_REPLACEMENT_STARTED=0
ORCA_POST_UPGRADE=
ORCA_ROLLBACK_BINARY_STAGED=
ORCA_ROLLBACK_VERSION_STAGED=
ORCA_ROLLBACK_HAS_VERSION=0
restart_after_rollback_error() {
exit_status=$?
trap - EXIT
set +e
if ((exit_status != 0 && ORCA_SERVICE_STOPPED)); then
recovery_ok=1
if ((${#ORCA_INSTALLED_RESTORE_DIRS[@]})); then
for profile_dir in "${ORCA_INSTALLED_RESTORE_DIRS[@]}"; do
if sudo test -d "/home/orca/.config/$profile_dir"; then
if ! sudo mv "/home/orca/.config/$profile_dir" \
"$ORCA_RESTORE/$profile_dir.failed"; then
recovery_ok=0
fi
fi
done
fi
if ((${#ORCA_MOVED_CURRENT_DIRS[@]})); then
for profile_dir in "${ORCA_MOVED_CURRENT_DIRS[@]}"; do
if sudo test -d "$ORCA_POST_UPGRADE/$profile_dir"; then
if ! sudo mv "$ORCA_POST_UPGRADE/$profile_dir" /home/orca/.config/; then
recovery_ok=0
fi
elif ! sudo test -d "/home/orca/.config/$profile_dir"; then
recovery_ok=0
fi
done
fi
if [[ -n "$ORCA_POST_UPGRADE" ]]; then
sudo rmdir "$ORCA_POST_UPGRADE" 2>/dev/null || true
fi
if ((ORCA_CURRENT_BINARY_MOVED)); then
if sudo test -f "$ORCA_CURRENT_BINARY"; then
if ! sudo mv -f "$ORCA_CURRENT_BINARY" /opt/orca/orca-linux.AppImage; then
recovery_ok=0
fi
elif ! sudo test -f /opt/orca/orca-linux.AppImage; then
recovery_ok=0
fi
fi
if ((ORCA_CURRENT_VERSION_MOVED)); then
if sudo test -f "$ORCA_CURRENT_VERSION"; then
if ! sudo mv -f "$ORCA_CURRENT_VERSION" /opt/orca/VERSION; then
recovery_ok=0
fi
elif ! sudo test -f /opt/orca/VERSION; then
recovery_ok=0
fi
elif ((ORCA_VERSION_REPLACEMENT_STARTED)); then
if ! sudo rm -f /opt/orca/VERSION; then
recovery_ok=0
fi
fi
if ((recovery_ok)); then
sudo systemctl start orca-serve.service || true
else
echo 'Rollback recovery failed; service remains stopped' >&2
fi
fi
if [[ -n "$ORCA_ROLLBACK_BINARY_STAGED" ]]; then
sudo rm -f -- "$ORCA_ROLLBACK_BINARY_STAGED"
fi
if [[ -n "$ORCA_ROLLBACK_VERSION_STAGED" ]]; then
sudo rm -f -- "$ORCA_ROLLBACK_VERSION_STAGED"
fi
sudo rm -rf -- "$ORCA_RESTORE"
exit "$exit_status"
}
trap restart_after_rollback_error EXIT
if [[ "$(sudo stat -c %d "$ORCA_RESTORE")" != \
"$(sudo stat -c %d /home/orca/.config)" ]]; then
echo 'Refusing rollback because staging and the Orca profile are on different filesystems' >&2
exit 1
fi
sudo tar xzf "$ORCA_ROLLBACK/profile.tgz" -C "$ORCA_RESTORE"
ORCA_RESTORE_DIRS=()
for profile_dir in orca Orca; do
if sudo test -L "$ORCA_RESTORE/$profile_dir"; then
echo "Rollback bundle contains a symlinked profile: $profile_dir" >&2
exit 1
fi
if sudo test -d "$ORCA_RESTORE/$profile_dir"; then
if [[ "$profile_dir" == Orca ]] && \
sudo test "$ORCA_RESTORE/orca" -ef "$ORCA_RESTORE/Orca"; then
continue
fi
ORCA_RESTORE_DIRS+=("$profile_dir")
fi
done
if ((${#ORCA_RESTORE_DIRS[@]} == 0)); then
echo "Rollback bundle has no Orca profile directories: $ORCA_ROLLBACK" >&2
exit 1
fi
for profile_dir in "${ORCA_RESTORE_DIRS[@]}"; do
sudo chown -R orca:orca "$ORCA_RESTORE/$profile_dir"
done
ORCA_ROLLBACK_STAMP=$(date +%F-%H%M%S-%N)
ORCA_ROLLBACK_BINARY_STAGED=/opt/orca/orca-linux.AppImage.rollback-staged-$ORCA_ROLLBACK_STAMP
sudo cp -a "$ORCA_ROLLBACK/orca-linux.AppImage" "$ORCA_ROLLBACK_BINARY_STAGED"
if sudo test -f "$ORCA_ROLLBACK/VERSION"; then
ORCA_ROLLBACK_HAS_VERSION=1
ORCA_ROLLBACK_VERSION_STAGED=/opt/orca/VERSION.rollback-staged-$ORCA_ROLLBACK_STAMP
sudo cp -a "$ORCA_ROLLBACK/VERSION" "$ORCA_ROLLBACK_VERSION_STAGED"
fi
ORCA_SERVICE_STOPPED=1
sudo systemctl stop orca-serve.service
# Preserve and replace only Orca-owned profile directories
ORCA_CURRENT_DIRS=()
for profile_dir in orca Orca; do
if sudo test -L "/home/orca/.config/$profile_dir"; then
echo "Refusing symlinked Orca profile: /home/orca/.config/$profile_dir" >&2
exit 1
fi
if sudo test -d "/home/orca/.config/$profile_dir"; then
if [[ "$profile_dir" == Orca ]] && \
sudo test /home/orca/.config/orca -ef /home/orca/.config/Orca; then
continue
fi
ORCA_CURRENT_DIRS+=("$profile_dir")
fi
done
ORCA_POST_UPGRADE=/home/orca/.config/orca-rollback-$ORCA_ROLLBACK_STAMP
sudo install -d -o orca -g orca -m 700 "$ORCA_POST_UPGRADE"
if ((${#ORCA_CURRENT_DIRS[@]})); then
for profile_dir in "${ORCA_CURRENT_DIRS[@]}"; do
ORCA_MOVED_CURRENT_DIRS+=("$profile_dir")
sudo mv "/home/orca/.config/$profile_dir" "$ORCA_POST_UPGRADE/"
done
fi
for profile_dir in "${ORCA_RESTORE_DIRS[@]}"; do
ORCA_INSTALLED_RESTORE_DIRS+=("$profile_dir")
sudo mv "$ORCA_RESTORE/$profile_dir" /home/orca/.config/
done
ORCA_CURRENT_BINARY=/opt/orca/orca-linux.AppImage.rollback-current-$ORCA_ROLLBACK_STAMP
ORCA_CURRENT_BINARY_MOVED=1
sudo mv /opt/orca/orca-linux.AppImage "$ORCA_CURRENT_BINARY"
sudo mv -f "$ORCA_ROLLBACK_BINARY_STAGED" /opt/orca/orca-linux.AppImage
ORCA_CURRENT_VERSION=/opt/orca/VERSION.rollback-current-$ORCA_ROLLBACK_STAMP
if sudo test -f /opt/orca/VERSION; then
ORCA_CURRENT_VERSION_MOVED=1
sudo mv /opt/orca/VERSION "$ORCA_CURRENT_VERSION"
fi
ORCA_VERSION_REPLACEMENT_STARTED=1
if ((ORCA_ROLLBACK_HAS_VERSION)); then
sudo mv -f "$ORCA_ROLLBACK_VERSION_STAGED" /opt/orca/VERSION
else
sudo rm -f /opt/orca/VERSION
fi
sudo systemctl start orca-serve.service
ORCA_SERVICE_STOPPED=0
sudo rm -rf -- "$ORCA_RESTORE"
trap - EXIT
```
Restoring the backup is required, not optional: swapping only the binary leaves
the newer `orca-data.json` in place, where an older build can discard state it
does not understand. Keep the pre-upgrade backup until the new version is proven
on your host. The `orca-rollback-*` directory inside `.config` is also retained
deliberately. The post-upgrade binary and version record are retained in
`/opt/orca` with the same `rollback-current-<timestamp>` suffix. Inspect these
artifacts and remove them according to your retention policy after the rollback
is resolved.
## Troubleshooting
- `dlopen(): error loading libfuse.so.2`: install `libfuse2`.
@@ -178,6 +611,9 @@ depend on an interactive shell profile.
`orca` user and that `/opt/orca` is readable by that user.
- Clients cannot connect: make sure `--pairing-address` is an address reachable
from the client, and make sure firewalls allow the selected `--port`.
- Service crash-loops right after an upgrade: use [Roll back](#roll-back) with
the pre-upgrade `.ready` bundle. Do not rerun the upgrade first; doing so would
make the crashing version the next rollback binary.
- Diagnosing other missing libraries: extract the AppImage without launching it
with `./orca-linux.AppImage --appimage-extract`, then run
`ldd squashfs-root/orca` to list any shared libraries the host is missing.