mirror of
https://github.com/stablyai/orca.git
synced 2026-10-22 16:03:47 +00:00
* fix(startup): stop a duplicate headless serve from crash-looping and leaking AppImage mounts A second Orca launch that loses the single-instance lock called app.quit() before `ready`. That quit is deferred, so the doomed process kept booting into Chromium's Linux display initialization, failed with "Missing X server or $DISPLAY", and died with SIGSEGV. systemd read that as a crash and restarted it forever; each restart re-mounted the AppImage and left the squashfuse mount behind, until the host hit the 1000-mount FUSE ceiling and every later launch failed. The lock-losing launch now calls app.exit(3), which terminates synchronously before any display init. Exit code 3 is a stable "another process already owns this userData profile" contract, and the documented systemd unit uses RestartPreventExitStatus=3 plus a real StartLimitIntervalSec/StartLimitBurst window so a permanently failing launch can no longer retry unbounded. Second-instance argv is now forwarded to the owner, and a duplicate `orca serve` no longer asks the live headless server to open a desktop window. Desktop activation for ordinary launches and macOS dock re-activation is unchanged. Closes #11935 * docs(headless): clear the start limit before the scripted service starts StartLimitIntervalSec=300/StartLimitBurst=5 rate-limits operator starts too, so after a crash-loop trips the burst systemd refuses a plain `systemctl start` for the rest of the window. The Upgrade and Roll back scripts run under `set -euo pipefail`, so that refusal aborted the rollback mid-flight and left the server down on the exact recovery path the doc prescribes. Both scripts (and their EXIT-trap recoveries) now run `systemctl reset-failed` first, the unit reference explains the interaction, and the crash-loop bullet points at it for manual starts. Co-authored-by: Orca <help@stably.ai> * test(startup): reproduce the #11935 duplicate-serve crash loop under real Electron The committed coverage for #11935 was source-text greps, so nothing gated the mechanism the fix rests on: pre-`ready` `app.quit()` is deferred, which is why the lock-losing headless `orca serve` kept booting into Linux display init. This runs two real Electron processes against one disposable profile. The duplicate executes the lock-loss gate's own `app.*` statement, lifted out of `src/main/index.ts`, so reverting to `app.quit()` fails the test. It also feeds the owner's real forwarded argv through `shouldActivateDesktopForSecondInstance`. Also record why the activation predicate matches `--serve` and not the `serve` subcommand: an AppImage launched as `orca serve` exits at the CLI redirect before requesting the lock. * test(startup): wait for the owner process to exit before removing its profile Windows holds the profile's handles for a beat after SIGKILL, so an immediate rmSync can fail with EBUSY/EPERM. Co-authored-by: Orca <help@stably.ai> * test(startup): pass the fixture marker path by env, not argv Chromium reorders argv and the duplicate's argv is itself under test, so a trailing positional was the wrong channel for it. Co-authored-by: Orca <help@stably.ai> * test(startup): only the activation case waits on the owner notification The exit-contract cases assert on the duplicate's own already-terminated process, so they should not block on cross-process delivery. Co-authored-by: Orca <help@stably.ai> * test(startup): drop the staged lock race, keep the real-Electron gate contract CI proved the two-process form cannot work on a display-less Linux runner: Chromium's ProcessSingleton needs the browser IO thread, which needs `ready`, which needs a display. The pre-`ready` owner looked stale and the duplicate took the lock (`expected [ 'DUPLICATE_WON_LOCK' ] to include 'DUPLICATE_LOST_LOCK'`). Lock acquisition and argv forwarding are already covered in single-instance-lock.test.ts. What only a real process can settle is what the loser does next, so that is all this file now runs -- display-independent. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai>
116 lines
4.8 KiB
TypeScript
116 lines
4.8 KiB
TypeScript
import { spawnSync } from 'node:child_process'
|
|
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
|
import { createRequire } from 'node:module'
|
|
import { tmpdir } from 'node:os'
|
|
import { join } from 'node:path'
|
|
import { afterAll, describe, expect, it } from 'vitest'
|
|
import { SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE } from './single-instance-lock'
|
|
|
|
// Why #11935: the lock-loss gate runs before Electron `ready`, where `app.quit()` is deferred, so a
|
|
// duplicate headless `orca serve` kept executing the rest of startup, reached Linux Ozone/X11 init
|
|
// with no display, died with SIGSEGV, and systemd restarted it until the leaked AppImage FUSE mounts
|
|
// hit the kernel's 1000-mount ceiling. This runs the gate's own termination statement, lifted out of
|
|
// `src/main/index.ts`, under the real Electron binary.
|
|
//
|
|
// Why not a live lock race: Chromium's Linux ProcessSingleton only answers a second process once the
|
|
// browser IO thread is up, which needs `ready` and therefore a display. On a display-less CI runner
|
|
// the "owner" is treated as stale and the duplicate takes the lock, so the race cannot be staged
|
|
// there. Lock acquisition and argv forwarding are covered in `single-instance-lock.test.ts`; what
|
|
// only a real process can settle is what the loser does next, which is what this file pins.
|
|
|
|
const electronBinary = createRequire(import.meta.url)('electron') as string
|
|
const LOCK_LOST = 'LOCK_LOST'
|
|
const CONTINUED_INTO_STARTUP = 'CONTINUED_INTO_STARTUP'
|
|
const REACHED_TAIL = 'REACHED_TAIL'
|
|
const MARKER_ENV = 'ORCA_LOCK_FIXTURE_MARKER'
|
|
|
|
const fixtureRoots: string[] = []
|
|
|
|
afterAll(() => {
|
|
for (const root of fixtureRoots) {
|
|
rmSync(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 })
|
|
}
|
|
})
|
|
|
|
/** The `app.*` call the shipped lock-loss gate executes, so a revert to `app.quit()` fails here. */
|
|
function readLockLossTermination(): string {
|
|
const source = readFileSync(join(process.cwd(), 'src/main/index.ts'), 'utf8')
|
|
const start = source.indexOf('if (!hasSingleInstanceLock) {')
|
|
expect(start).toBeGreaterThanOrEqual(0)
|
|
const end = source.indexOf('\n}', start)
|
|
expect(end).toBeGreaterThan(start)
|
|
|
|
return source
|
|
.slice(start, end)
|
|
.split('\n')
|
|
.map((line) => line.trim())
|
|
.filter((line) => line.startsWith('app.'))
|
|
.join('\n')
|
|
}
|
|
|
|
function buildFixtureMain(termination: string): string {
|
|
return [
|
|
`const { app } = require('electron')`,
|
|
`const { appendFileSync } = require('node:fs')`,
|
|
// Why: the marker path travels by env — Chromium rewrites argv before the main script sees it.
|
|
`const marker = process.env.${MARKER_ENV}`,
|
|
`const mark = (name) => appendFileSync(marker, name + '\\n')`,
|
|
`const SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE = ${SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE}`,
|
|
`mark('${LOCK_LOST}')`,
|
|
termination,
|
|
`mark('${CONTINUED_INTO_STARTUP}')`,
|
|
// Why: stand in for the rest of `src/main/index.ts`, which on the reported host was display init.
|
|
`mark('${REACHED_TAIL}')`,
|
|
`process.exit(0)`
|
|
].join('\n')
|
|
}
|
|
|
|
type FixtureRun = { status: number | null; markers: string[] }
|
|
|
|
function runLockLossGate(termination: string): FixtureRun {
|
|
const root = mkdtempSync(join(tmpdir(), 'orca-lock-loss-'))
|
|
fixtureRoots.push(root)
|
|
const dir = join(root, 'fixture')
|
|
const marker = join(root, 'markers.log')
|
|
mkdirSync(dir, { recursive: true })
|
|
writeFileSync(
|
|
join(dir, 'package.json'),
|
|
'{ "name": "orca-lock-loss-fixture", "main": "main.js" }'
|
|
)
|
|
writeFileSync(join(dir, 'main.js'), buildFixtureMain(termination))
|
|
writeFileSync(marker, '')
|
|
|
|
const result = spawnSync(electronBinary, [dir, '--no-sandbox'], {
|
|
stdio: 'ignore',
|
|
timeout: 60_000,
|
|
env: { ...process.env, [MARKER_ENV]: marker }
|
|
})
|
|
expect(result.error).toBeUndefined()
|
|
|
|
return {
|
|
status: result.status,
|
|
markers: readFileSync(marker, 'utf8').split('\n').filter(Boolean)
|
|
}
|
|
}
|
|
|
|
describe('#11935 pre-ready lock-loss termination under real Electron', () => {
|
|
it('stops the duplicate launch before any further startup runs, with the already-running code', () => {
|
|
const termination = readLockLossTermination()
|
|
// Why: an empty slice would let the fixture fall through to its own exit and pass vacuously.
|
|
expect(termination).not.toBe('')
|
|
|
|
const run = runLockLossGate(termination)
|
|
|
|
expect(run.markers).toEqual([LOCK_LOST])
|
|
expect(run.status).toBe(SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE)
|
|
}, 90_000)
|
|
|
|
it('reproduces the deferred graceful quit that let the doomed launch keep booting', () => {
|
|
const run = runLockLossGate('app.quit()')
|
|
|
|
// Why: pins the Electron semantic the fix rests on — pre-`ready` `quit()` schedules, it does not stop.
|
|
expect(run.markers).toEqual([LOCK_LOST, CONTINUED_INTO_STARTUP, REACHED_TAIL])
|
|
expect(run.status).not.toBe(SINGLE_INSTANCE_ALREADY_RUNNING_EXIT_CODE)
|
|
}, 90_000)
|
|
})
|