Files
orca/src/main/startup/single-instance-lock-exit.electron.test.ts
T
NeilandOrca d8e5944b60 Stop a duplicate headless orca serve from crash-looping and exhausting AppImage FUSE mounts (#12212)
* fix(startup): stop a duplicate headless serve from crash-looping and leaking AppImage mounts

A second Orca launch that loses the single-instance lock called app.quit()
before `ready`. That quit is deferred, so the doomed process kept booting into
Chromium's Linux display initialization, failed with "Missing X server or
$DISPLAY", and died with SIGSEGV. systemd read that as a crash and restarted it
forever; each restart re-mounted the AppImage and left the squashfuse mount
behind, until the host hit the 1000-mount FUSE ceiling and every later launch
failed.

The lock-losing launch now calls app.exit(3), which terminates synchronously
before any display init. Exit code 3 is a stable "another process already owns
this userData profile" contract, and the documented systemd unit uses
RestartPreventExitStatus=3 plus a real StartLimitIntervalSec/StartLimitBurst
window so a permanently failing launch can no longer retry unbounded.

Second-instance argv is now forwarded to the owner, and a duplicate `orca serve`
no longer asks the live headless server to open a desktop window. Desktop
activation for ordinary launches and macOS dock re-activation is unchanged.

Closes #11935

* docs(headless): clear the start limit before the scripted service starts

StartLimitIntervalSec=300/StartLimitBurst=5 rate-limits operator starts too, so
after a crash-loop trips the burst systemd refuses a plain `systemctl start` for
the rest of the window. The Upgrade and Roll back scripts run under
`set -euo pipefail`, so that refusal aborted the rollback mid-flight and left the
server down on the exact recovery path the doc prescribes.

Both scripts (and their EXIT-trap recoveries) now run `systemctl reset-failed`
first, the unit reference explains the interaction, and the crash-loop bullet
points at it for manual starts.

Co-authored-by: Orca <help@stably.ai>

* test(startup): reproduce the #11935 duplicate-serve crash loop under real Electron

The committed coverage for #11935 was source-text greps, so nothing gated the
mechanism the fix rests on: pre-`ready` `app.quit()` is deferred, which is why
the lock-losing headless `orca serve` kept booting into Linux display init.

This runs two real Electron processes against one disposable profile. The
duplicate executes the lock-loss gate's own `app.*` statement, lifted out of
`src/main/index.ts`, so reverting to `app.quit()` fails the test. It also feeds
the owner's real forwarded argv through `shouldActivateDesktopForSecondInstance`.

Also record why the activation predicate matches `--serve` and not the `serve`
subcommand: an AppImage launched as `orca serve` exits at the CLI redirect
before requesting the lock.

* test(startup): wait for the owner process to exit before removing its profile

Windows holds the profile's handles for a beat after SIGKILL, so an immediate
rmSync can fail with EBUSY/EPERM.

Co-authored-by: Orca <help@stably.ai>

* test(startup): pass the fixture marker path by env, not argv

Chromium reorders argv and the duplicate's argv is itself under test, so a
trailing positional was the wrong channel for it.

Co-authored-by: Orca <help@stably.ai>

* test(startup): only the activation case waits on the owner notification

The exit-contract cases assert on the duplicate's own already-terminated
process, so they should not block on cross-process delivery.

Co-authored-by: Orca <help@stably.ai>

* test(startup): drop the staged lock race, keep the real-Electron gate contract

CI proved the two-process form cannot work on a display-less Linux runner:
Chromium's ProcessSingleton needs the browser IO thread, which needs `ready`,
which needs a display. The pre-`ready` owner looked stale and the duplicate took
the lock (`expected [ 'DUPLICATE_WON_LOCK' ] to include 'DUPLICATE_LOST_LOCK'`).

Lock acquisition and argv forwarding are already covered in
single-instance-lock.test.ts. What only a real process can settle is what the
loser does next, so that is all this file now runs -- display-independent.

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-08-03 23:34:22 -07:00

116 lines
4.8 KiB
TypeScript

import { spawnSync } from 'node:child_process'
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { createRequire } from 'node:module'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterAll, describe, expect, it } from 'vitest'
import { SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE } from './single-instance-lock'
// Why #11935: the lock-loss gate runs before Electron `ready`, where `app.quit()` is deferred, so a
// duplicate headless `orca serve` kept executing the rest of startup, reached Linux Ozone/X11 init
// with no display, died with SIGSEGV, and systemd restarted it until the leaked AppImage FUSE mounts
// hit the kernel's 1000-mount ceiling. This runs the gate's own termination statement, lifted out of
// `src/main/index.ts`, under the real Electron binary.
//
// Why not a live lock race: Chromium's Linux ProcessSingleton only answers a second process once the
// browser IO thread is up, which needs `ready` and therefore a display. On a display-less CI runner
// the "owner" is treated as stale and the duplicate takes the lock, so the race cannot be staged
// there. Lock acquisition and argv forwarding are covered in `single-instance-lock.test.ts`; what
// only a real process can settle is what the loser does next, which is what this file pins.
const electronBinary = createRequire(import.meta.url)('electron') as string
const LOCK_LOST = 'LOCK_LOST'
const CONTINUED_INTO_STARTUP = 'CONTINUED_INTO_STARTUP'
const REACHED_TAIL = 'REACHED_TAIL'
const MARKER_ENV = 'ORCA_LOCK_FIXTURE_MARKER'
const fixtureRoots: string[] = []
afterAll(() => {
for (const root of fixtureRoots) {
rmSync(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 })
}
})
/** The `app.*` call the shipped lock-loss gate executes, so a revert to `app.quit()` fails here. */
function readLockLossTermination(): string {
const source = readFileSync(join(process.cwd(), 'src/main/index.ts'), 'utf8')
const start = source.indexOf('if (!hasSingleInstanceLock) {')
expect(start).toBeGreaterThanOrEqual(0)
const end = source.indexOf('\n}', start)
expect(end).toBeGreaterThan(start)
return source
.slice(start, end)
.split('\n')
.map((line) => line.trim())
.filter((line) => line.startsWith('app.'))
.join('\n')
}
function buildFixtureMain(termination: string): string {
return [
`const { app } = require('electron')`,
`const { appendFileSync } = require('node:fs')`,
// Why: the marker path travels by env — Chromium rewrites argv before the main script sees it.
`const marker = process.env.${MARKER_ENV}`,
`const mark = (name) => appendFileSync(marker, name + '\\n')`,
`const SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE = ${SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE}`,
`mark('${LOCK_LOST}')`,
termination,
`mark('${CONTINUED_INTO_STARTUP}')`,
// Why: stand in for the rest of `src/main/index.ts`, which on the reported host was display init.
`mark('${REACHED_TAIL}')`,
`process.exit(0)`
].join('\n')
}
type FixtureRun = { status: number | null; markers: string[] }
function runLockLossGate(termination: string): FixtureRun {
const root = mkdtempSync(join(tmpdir(), 'orca-lock-loss-'))
fixtureRoots.push(root)
const dir = join(root, 'fixture')
const marker = join(root, 'markers.log')
mkdirSync(dir, { recursive: true })
writeFileSync(
join(dir, 'package.json'),
'{ "name": "orca-lock-loss-fixture", "main": "main.js" }'
)
writeFileSync(join(dir, 'main.js'), buildFixtureMain(termination))
writeFileSync(marker, '')
const result = spawnSync(electronBinary, [dir, '--no-sandbox'], {
stdio: 'ignore',
timeout: 60_000,
env: { ...process.env, [MARKER_ENV]: marker }
})
expect(result.error).toBeUndefined()
return {
status: result.status,
markers: readFileSync(marker, 'utf8').split('\n').filter(Boolean)
}
}
describe('#11935 pre-ready lock-loss termination under real Electron', () => {
it('stops the duplicate launch before any further startup runs, with the already-running code', () => {
const termination = readLockLossTermination()
// Why: an empty slice would let the fixture fall through to its own exit and pass vacuously.
expect(termination).not.toBe('')
const run = runLockLossGate(termination)
expect(run.markers).toEqual([LOCK_LOST])
expect(run.status).toBe(SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE)
}, 90_000)
it('reproduces the deferred graceful quit that let the doomed launch keep booting', () => {
const run = runLockLossGate('app.quit()')
// Why: pins the Electron semantic the fix rests on — pre-`ready` `quit()` schedules, it does not stop.
expect(run.markers).toEqual([LOCK_LOST, CONTINUED_INTO_STARTUP, REACHED_TAIL])
expect(run.status).not.toBe(SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE)
}, 90_000)
})