diff --git a/docs/reference/headless-linux-server.md b/docs/reference/headless-linux-server.md index e6267e84f8a..316f0759ea0 100644 --- a/docs/reference/headless-linux-server.md +++ b/docs/reference/headless-linux-server.md @@ -15,7 +15,7 @@ Install the AppImage runtime dependency and Xvfb: ```bash sudo apt-get update -sudo apt-get install -y curl libfuse2 xvfb +sudo apt-get install -y curl file libfuse2 xvfb ``` Download and make the AppImage executable: @@ -56,11 +56,14 @@ The command prints the runtime endpoint and pairing URL. Stop it with `Ctrl+C`. ## Systemd Service Create a dedicated service user and install directory. Run the service as this -user instead of root so the AppImage can keep Chromium's sandbox enabled. +user instead of root so the AppImage can keep Chromium's sandbox enabled. Keep +the install directory root-owned: the service needs to read and execute the +AppImage, but must not be able to replace it or the rollback artifacts. ```bash sudo useradd --system --create-home --shell /usr/sbin/nologin orca -sudo chown -R orca:orca /opt/orca +sudo chown root:root /opt/orca /opt/orca/orca-linux.AppImage +sudo chmod 755 /opt/orca /opt/orca/orca-linux.AppImage ``` For most hosts, one `orca serve` service is enough because Orca starts Xvfb on @@ -165,6 +168,436 @@ If you later install the desktop CLI from Orca settings, use that CLI for normal shell workflows. Keep the AppImage path in systemd so service restarts do not depend on an interactive shell profile. +## Upgrade + +`orca serve` never updates itself. In headless mode Orca wires up no auto-updater +at all — the built-in updater only runs in the desktop GUI, and no paired mobile +or web client can trigger it remotely. Upgrading is always a deliberate step: +replace the AppImage and restart the service. + +Two facts make this safe and predictable: + +- **State lives in the service user's home, not next to the binary.** Persisted + data is under `/home/orca/.config/` (Orca uses both an `orca` and an `Orca` + directory there), fully independent of `/opt/orca/orca-linux.AppImage`. + Replacing the binary never touches projects, worktree metadata, terminal + history, orchestration state, or paired-device keys — so mobile and web + clients reconnect after an upgrade without re-pairing. +- **New builds migrate old state on load.** Orca loads older `orca-data.json` + state into the current schema and writes it back in the current shape, so a + forward upgrade needs no manual data step. + +Rolling back is the case that needs care — see [Roll back](#roll-back). + +### Record the version you deploy + +Orca has no headless version command: there is no `--version` flag or `version` +subcommand, and `orca serve` prints only its endpoint. Choose a release tag +explicitly instead of following the `latest` URL, and record it next to the +binary so upgrades are auditable. The steps below keep that record in +`/opt/orca/VERSION`. + +### Upgrade steps + +Never download straight onto `/opt/orca/orca-linux.AppImage`. The AppImage is +FUSE-mounted, so overwriting it in place while the service runs can crash or +corrupt the live process — and even with the service stopped, a failed or partial +download would clobber the working binary. Instead download to a temporary name +on the same filesystem, verify it, then swap it in with an atomic rename. + +Check capacity before starting: + +```bash +sudo chown root:root /opt/orca +sudo chmod 755 /opt/orca +sudo test ! -L /opt/orca/orca-linux.AppImage +sudo chown root:root /opt/orca/orca-linux.AppImage +sudo chmod 755 /opt/orca/orca-linux.AppImage +# Clear predictable staging names left by an older attempt after locking the directory +sudo rm -f /opt/orca/orca-linux.AppImage.new /opt/orca/VERSION.new \ + /opt/orca/orca-linux.AppImage.recovering /opt/orca/VERSION.recovering +sudo du -sh /home/orca/.config +df -h /opt/orca /home/orca +``` + +`/opt/orca` needs room for the compressed Orca profile archive, the staged +build, and the rollback binary. A rollback extracts the old profile and preserves +the post-upgrade Orca profile directories, so `/home` needs room for both copies. + +Run the following block as one Bash script so its fail-fast and recovery traps +remain active for the whole operation: + +```bash +set -euo pipefail + +# Replace this example with the release tag you intend to deploy +ORCA_VERSION=v1.4.147 + +# Select the release asset on the server where Orca runs +case "$(uname -m)" in + x86_64) + ORCA_ASSET=orca-linux.AppImage + ORCA_FILE_MACHINE=x86-64 + ;; + aarch64 | arm64) + ORCA_ASSET=orca-linux-arm64.AppImage + ORCA_FILE_MACHINE='ARM aarch64' + ;; + *) + echo "Unsupported architecture: $(uname -m)" >&2 + exit 1 + ;; +esac + +ORCA_ROLLBACK_NEW= +ORCA_ROLLBACK= +ORCA_SERVICE_STOPPED=0 +ORCA_BINARY_PROMOTED=0 +recover_failed_upgrade() { + exit_status=$? + trap - EXIT + set +e + if ((exit_status != 0)); then + sudo rm -f /opt/orca/orca-linux.AppImage.new /opt/orca/VERSION.new \ + /opt/orca/orca-linux.AppImage.recovering /opt/orca/VERSION.recovering + fi + if ((exit_status != 0)) && [[ -n "$ORCA_ROLLBACK_NEW" ]] && \ + sudo test -d "$ORCA_ROLLBACK_NEW"; then + sudo rm -rf -- "$ORCA_ROLLBACK_NEW" + fi + if ((exit_status != 0 && ORCA_SERVICE_STOPPED)); then + recovery_ok=1 + if ((ORCA_BINARY_PROMOTED)); then + if ! sudo cp -a "$ORCA_ROLLBACK/orca-linux.AppImage" \ + /opt/orca/orca-linux.AppImage.recovering || \ + ! sudo mv -f /opt/orca/orca-linux.AppImage.recovering \ + /opt/orca/orca-linux.AppImage; then + recovery_ok=0 + fi + if sudo test -f "$ORCA_ROLLBACK/VERSION"; then + if ! sudo cp -a "$ORCA_ROLLBACK/VERSION" /opt/orca/VERSION.recovering || \ + ! sudo mv -f /opt/orca/VERSION.recovering /opt/orca/VERSION; then + recovery_ok=0 + fi + elif ! sudo rm -f /opt/orca/VERSION; then + recovery_ok=0 + fi + fi + sudo rm -f /opt/orca/orca-linux.AppImage.recovering \ + /opt/orca/VERSION.recovering + if ((recovery_ok)); then + sudo systemctl start orca-serve.service || true + else + echo 'Upgrade recovery failed; service remains stopped' >&2 + fi + fi + exit "$exit_status" +} +trap recover_failed_upgrade EXIT + +# 1. Stage and verify the new build while the server stays online +sudo curl -fL --retry 3 "https://github.com/stablyai/orca/releases/download/${ORCA_VERSION}/${ORCA_ASSET}" \ + -o /opt/orca/orca-linux.AppImage.new +sudo chown root:root /opt/orca/orca-linux.AppImage.new +sudo chmod 755 /opt/orca/orca-linux.AppImage.new + +# Both checks must match; either grep stops this fail-fast block otherwise +ORCA_FILE_INFO=$(LC_ALL=C file /opt/orca/orca-linux.AppImage.new) +grep 'ELF .* executable' <<<"$ORCA_FILE_INFO" +grep -F "$ORCA_FILE_MACHINE" <<<"$ORCA_FILE_INFO" + +# 2. Assemble the prior binary and version in a root-only rollback bundle +ORCA_ROLLBACK_BASE=/opt/orca/orca-rollback-$(date +%F-%H%M%S-%N) +ORCA_ROLLBACK_NEW=${ORCA_ROLLBACK_BASE}.new +ORCA_ROLLBACK=${ORCA_ROLLBACK_BASE}.ready +sudo install -d -m 700 "$ORCA_ROLLBACK_NEW" +sudo cp -a /opt/orca/orca-linux.AppImage "$ORCA_ROLLBACK_NEW/orca-linux.AppImage" +if sudo test -f /opt/orca/VERSION; then + sudo cp -a /opt/orca/VERSION "$ORCA_ROLLBACK_NEW/VERSION" +fi + +# Stage the new version record before the stop window +printf '%s\n' "$ORCA_VERSION" | sudo tee /opt/orca/VERSION.new >/dev/null +sudo chown root:root /opt/orca/VERSION.new +sudo chmod 644 /opt/orca/VERSION.new + +# 3. Stop the server so the profile backup is consistent +ORCA_SERVICE_STOPPED=1 +sudo systemctl stop orca-serve.service + +# Add only Orca-owned profile directories, then publish the complete bundle +ORCA_PROFILE_DIRS=() +for profile_dir in orca Orca; do + if sudo test -L "/home/orca/.config/$profile_dir"; then + echo "Refusing symlinked Orca profile: /home/orca/.config/$profile_dir" >&2 + exit 1 + fi + if sudo test -d "/home/orca/.config/$profile_dir"; then + if [[ "$profile_dir" == Orca ]] && \ + sudo test /home/orca/.config/orca -ef /home/orca/.config/Orca; then + continue + fi + ORCA_PROFILE_DIRS+=("$profile_dir") + fi +done +if ((${#ORCA_PROFILE_DIRS[@]} == 0)); then + echo 'No Orca profile directory found under /home/orca/.config' >&2 + exit 1 +fi +sudo tar czf "$ORCA_ROLLBACK_NEW/profile.tgz" \ + -C /home/orca/.config "${ORCA_PROFILE_DIRS[@]}" +sudo chmod 600 "$ORCA_ROLLBACK_NEW/profile.tgz" +sudo mv "$ORCA_ROLLBACK_NEW" "$ORCA_ROLLBACK" + +# 4. Atomically replace the binary and version record, then start +ORCA_BINARY_PROMOTED=1 +sudo mv -f /opt/orca/orca-linux.AppImage.new /opt/orca/orca-linux.AppImage +sudo mv -f /opt/orca/VERSION.new /opt/orca/VERSION +sudo systemctl start orca-serve.service +ORCA_SERVICE_STOPPED=0 +trap - EXIT +``` + +The profile archive created in step 3 captures both Orca profile directory names +when present without rewinding unrelated tools under `/home/orca/.config`. The +`.ready` suffix is published only after the prior binary, version record, and +profile archive are complete. If you run the managed Xvfb unit, only +`orca-serve.service` needs restarting — leave `orca-xvfb.service` running. + +### Verify + +```bash +sudo journalctl -u orca-serve.service -f +``` + +A healthy start prints `Orca server ready: ws://0.0.0.0:6768` (with your port). +Confirm a client reconnects before you discard the backup. The timestamped +rollback bundles are not pruned automatically. After the new version satisfies +your retention policy, select and inspect the newest complete bundle before +removing it: + +```bash +shopt -s nullglob +ORCA_ROLLBACK_SETS=(/opt/orca/orca-rollback-*.ready) +((${#ORCA_ROLLBACK_SETS[@]} > 0)) +ORCA_ROLLBACK=${ORCA_ROLLBACK_SETS[${#ORCA_ROLLBACK_SETS[@]} - 1]} +printf 'Removing rollback bundle: %s\n' "$ORCA_ROLLBACK" +sudo test -d "$ORCA_ROLLBACK" +sudo rm -rf -- "$ORCA_ROLLBACK" +``` + +Each `.ready` directory is a self-contained rollback generation; never combine +files from different bundles. + +### Roll back + +A rollback is **not** binary-only safe. Once a newer build has started, it can +rewrite `orca-data.json` in the current schema. If an older build then writes +that file, it can discard fields it does not recognize. The rolling +`orca-data.json.bak.*` files are corruption-recovery snapshots, not a dedicated +pre-upgrade copy, and normal writes can rotate them away. To roll back cleanly, +restore the backup from step 3 **and** swap the binary back. Run this block as one +Bash script: + +```bash +set -euo pipefail + +# Select and validate one complete generation before taking the service offline +shopt -s nullglob +ORCA_ROLLBACK_SETS=(/opt/orca/orca-rollback-*.ready) +((${#ORCA_ROLLBACK_SETS[@]} > 0)) +ORCA_ROLLBACK=${ORCA_ROLLBACK_SETS[${#ORCA_ROLLBACK_SETS[@]} - 1]} +sudo test -f "$ORCA_ROLLBACK/orca-linux.AppImage" +sudo tar tzf "$ORCA_ROLLBACK/profile.tgz" >/dev/null + +# Extract and validate the old profile while the current server stays online +sudo test ! -L /home +ORCA_HOME_OWNER=$(sudo stat -c %u /home) +ORCA_HOME_MODE=$(sudo stat -c %a /home) +if [[ "$ORCA_HOME_OWNER" != 0 ]] || ((8#$ORCA_HOME_MODE & 0022)) || \ + sudo -u orca test -w /home; then + echo 'Refusing rollback because /home is not root-controlled' >&2 + exit 1 +fi +ORCA_RESTORE=$(sudo mktemp -d /home/.orca-restore.XXXXXX) +ORCA_SERVICE_STOPPED=0 +ORCA_MOVED_CURRENT_DIRS=() +ORCA_INSTALLED_RESTORE_DIRS=() +ORCA_CURRENT_BINARY_MOVED=0 +ORCA_CURRENT_VERSION_MOVED=0 +ORCA_VERSION_REPLACEMENT_STARTED=0 +ORCA_POST_UPGRADE= +ORCA_ROLLBACK_BINARY_STAGED= +ORCA_ROLLBACK_VERSION_STAGED= +ORCA_ROLLBACK_HAS_VERSION=0 +restart_after_rollback_error() { + exit_status=$? + trap - EXIT + set +e + if ((exit_status != 0 && ORCA_SERVICE_STOPPED)); then + recovery_ok=1 + if ((${#ORCA_INSTALLED_RESTORE_DIRS[@]})); then + for profile_dir in "${ORCA_INSTALLED_RESTORE_DIRS[@]}"; do + if sudo test -d "/home/orca/.config/$profile_dir"; then + if ! sudo mv "/home/orca/.config/$profile_dir" \ + "$ORCA_RESTORE/$profile_dir.failed"; then + recovery_ok=0 + fi + fi + done + fi + if ((${#ORCA_MOVED_CURRENT_DIRS[@]})); then + for profile_dir in "${ORCA_MOVED_CURRENT_DIRS[@]}"; do + if sudo test -d "$ORCA_POST_UPGRADE/$profile_dir"; then + if ! sudo mv "$ORCA_POST_UPGRADE/$profile_dir" /home/orca/.config/; then + recovery_ok=0 + fi + elif ! sudo test -d "/home/orca/.config/$profile_dir"; then + recovery_ok=0 + fi + done + fi + if [[ -n "$ORCA_POST_UPGRADE" ]]; then + sudo rmdir "$ORCA_POST_UPGRADE" 2>/dev/null || true + fi + if ((ORCA_CURRENT_BINARY_MOVED)); then + if sudo test -f "$ORCA_CURRENT_BINARY"; then + if ! sudo mv -f "$ORCA_CURRENT_BINARY" /opt/orca/orca-linux.AppImage; then + recovery_ok=0 + fi + elif ! sudo test -f /opt/orca/orca-linux.AppImage; then + recovery_ok=0 + fi + fi + if ((ORCA_CURRENT_VERSION_MOVED)); then + if sudo test -f "$ORCA_CURRENT_VERSION"; then + if ! sudo mv -f "$ORCA_CURRENT_VERSION" /opt/orca/VERSION; then + recovery_ok=0 + fi + elif ! sudo test -f /opt/orca/VERSION; then + recovery_ok=0 + fi + elif ((ORCA_VERSION_REPLACEMENT_STARTED)); then + if ! sudo rm -f /opt/orca/VERSION; then + recovery_ok=0 + fi + fi + if ((recovery_ok)); then + sudo systemctl start orca-serve.service || true + else + echo 'Rollback recovery failed; service remains stopped' >&2 + fi + fi + if [[ -n "$ORCA_ROLLBACK_BINARY_STAGED" ]]; then + sudo rm -f -- "$ORCA_ROLLBACK_BINARY_STAGED" + fi + if [[ -n "$ORCA_ROLLBACK_VERSION_STAGED" ]]; then + sudo rm -f -- "$ORCA_ROLLBACK_VERSION_STAGED" + fi + sudo rm -rf -- "$ORCA_RESTORE" + exit "$exit_status" +} +trap restart_after_rollback_error EXIT + +if [[ "$(sudo stat -c %d "$ORCA_RESTORE")" != \ + "$(sudo stat -c %d /home/orca/.config)" ]]; then + echo 'Refusing rollback because staging and the Orca profile are on different filesystems' >&2 + exit 1 +fi +sudo tar xzf "$ORCA_ROLLBACK/profile.tgz" -C "$ORCA_RESTORE" +ORCA_RESTORE_DIRS=() +for profile_dir in orca Orca; do + if sudo test -L "$ORCA_RESTORE/$profile_dir"; then + echo "Rollback bundle contains a symlinked profile: $profile_dir" >&2 + exit 1 + fi + if sudo test -d "$ORCA_RESTORE/$profile_dir"; then + if [[ "$profile_dir" == Orca ]] && \ + sudo test "$ORCA_RESTORE/orca" -ef "$ORCA_RESTORE/Orca"; then + continue + fi + ORCA_RESTORE_DIRS+=("$profile_dir") + fi +done +if ((${#ORCA_RESTORE_DIRS[@]} == 0)); then + echo "Rollback bundle has no Orca profile directories: $ORCA_ROLLBACK" >&2 + exit 1 +fi +for profile_dir in "${ORCA_RESTORE_DIRS[@]}"; do + sudo chown -R orca:orca "$ORCA_RESTORE/$profile_dir" +done + +ORCA_ROLLBACK_STAMP=$(date +%F-%H%M%S-%N) +ORCA_ROLLBACK_BINARY_STAGED=/opt/orca/orca-linux.AppImage.rollback-staged-$ORCA_ROLLBACK_STAMP +sudo cp -a "$ORCA_ROLLBACK/orca-linux.AppImage" "$ORCA_ROLLBACK_BINARY_STAGED" +if sudo test -f "$ORCA_ROLLBACK/VERSION"; then + ORCA_ROLLBACK_HAS_VERSION=1 + ORCA_ROLLBACK_VERSION_STAGED=/opt/orca/VERSION.rollback-staged-$ORCA_ROLLBACK_STAMP + sudo cp -a "$ORCA_ROLLBACK/VERSION" "$ORCA_ROLLBACK_VERSION_STAGED" +fi + +ORCA_SERVICE_STOPPED=1 +sudo systemctl stop orca-serve.service + +# Preserve and replace only Orca-owned profile directories +ORCA_CURRENT_DIRS=() +for profile_dir in orca Orca; do + if sudo test -L "/home/orca/.config/$profile_dir"; then + echo "Refusing symlinked Orca profile: /home/orca/.config/$profile_dir" >&2 + exit 1 + fi + if sudo test -d "/home/orca/.config/$profile_dir"; then + if [[ "$profile_dir" == Orca ]] && \ + sudo test /home/orca/.config/orca -ef /home/orca/.config/Orca; then + continue + fi + ORCA_CURRENT_DIRS+=("$profile_dir") + fi +done +ORCA_POST_UPGRADE=/home/orca/.config/orca-rollback-$ORCA_ROLLBACK_STAMP +sudo install -d -o orca -g orca -m 700 "$ORCA_POST_UPGRADE" +if ((${#ORCA_CURRENT_DIRS[@]})); then + for profile_dir in "${ORCA_CURRENT_DIRS[@]}"; do + ORCA_MOVED_CURRENT_DIRS+=("$profile_dir") + sudo mv "/home/orca/.config/$profile_dir" "$ORCA_POST_UPGRADE/" + done +fi +for profile_dir in "${ORCA_RESTORE_DIRS[@]}"; do + ORCA_INSTALLED_RESTORE_DIRS+=("$profile_dir") + sudo mv "$ORCA_RESTORE/$profile_dir" /home/orca/.config/ +done + +ORCA_CURRENT_BINARY=/opt/orca/orca-linux.AppImage.rollback-current-$ORCA_ROLLBACK_STAMP +ORCA_CURRENT_BINARY_MOVED=1 +sudo mv /opt/orca/orca-linux.AppImage "$ORCA_CURRENT_BINARY" +sudo mv -f "$ORCA_ROLLBACK_BINARY_STAGED" /opt/orca/orca-linux.AppImage + +ORCA_CURRENT_VERSION=/opt/orca/VERSION.rollback-current-$ORCA_ROLLBACK_STAMP +if sudo test -f /opt/orca/VERSION; then + ORCA_CURRENT_VERSION_MOVED=1 + sudo mv /opt/orca/VERSION "$ORCA_CURRENT_VERSION" +fi +ORCA_VERSION_REPLACEMENT_STARTED=1 +if ((ORCA_ROLLBACK_HAS_VERSION)); then + sudo mv -f "$ORCA_ROLLBACK_VERSION_STAGED" /opt/orca/VERSION +else + sudo rm -f /opt/orca/VERSION +fi +sudo systemctl start orca-serve.service +ORCA_SERVICE_STOPPED=0 +sudo rm -rf -- "$ORCA_RESTORE" +trap - EXIT +``` + +Restoring the backup is required, not optional: swapping only the binary leaves +the newer `orca-data.json` in place, where an older build can discard state it +does not understand. Keep the pre-upgrade backup until the new version is proven +on your host. The `orca-rollback-*` directory inside `.config` is also retained +deliberately. The post-upgrade binary and version record are retained in +`/opt/orca` with the same `rollback-current-` suffix. Inspect these +artifacts and remove them according to your retention policy after the rollback +is resolved. + ## Troubleshooting - `dlopen(): error loading libfuse.so.2`: install `libfuse2`. @@ -178,6 +611,9 @@ depend on an interactive shell profile. `orca` user and that `/opt/orca` is readable by that user. - Clients cannot connect: make sure `--pairing-address` is an address reachable from the client, and make sure firewalls allow the selected `--port`. +- Service crash-loops right after an upgrade: use [Roll back](#roll-back) with + the pre-upgrade `.ready` bundle. Do not rerun the upgrade first; doing so would + make the crashing version the next rollback binary. - Diagnosing other missing libraries: extract the AppImage without launching it with `./orca-linux.AppImage --appimage-extract`, then run `ldd squashfs-root/orca` to list any shared libraries the host is missing.