Files
orca/src/shared/orcad-managed-runtime.ts
T
NeilandJesse Sanford 7cafdb072b feat(ssh): Recover server for a managed host refused on a stale lock, and surface that refusal in seconds (#32951)
* feat(orcad): run --reclaim-stale-lock on a managed host's active slot

The desktop can now dry-run and confirm the stale instance-lock recovery
on a Linux managed host, serialized with launches, then start the server
again and reconnect the way the restart action does.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* feat(settings): offer Recover server for a managed host refused on a stale lock

When a managed orcad start names --reclaim-stale-lock, the SSH host row
offers Recover server. The dialog dry-runs the check, shows the host's
evidence, and clears the lock only after the user confirms no other
machine sharing the home runs Orca.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* feat(mobile): point a host refused on a stale lock to the desktop's Recover server

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(settings): mount the Recover server dialog per opening

A fresh mount starts each opening from a new check instead of resetting
state in an effect.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): end the readiness wait once the host proves the launch exited

A managed orcad refused at start (such as on a possibly stale instance
lock) left the client waiting out the full readiness deadline. Each
host-side wait now also checks the launch's PID file with kill -0 and
ends early on a proven exit, so the caller reports the same pending
readiness, with the orcad.log tail, within a poll. No PID, a denied
probe or a reused PID keeps the old deadline.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): refuse a managed slot without --reclaim-stale-lock before running it

An older orcad ignores the flag and starts serving in the foreground. The
same remote exec now checks that the slot's server bundle names the flag
and answers unsupported otherwise, without invoking orcad.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(settings): show the manual stale-lock command as literal mono text

The rendered dialog ran the command into the sentence, so it was hard to
read or copy. It now sits in its own mono block, like Details.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(settings): offer only a manual delete when the host can't reclaim, and format the lock's age

An unsupported host was told to run the very command it lacks. Main now
names the lock path for that case, and the dialog shows only a copyable
rm of it with the shared-home caution. The lock's age uses the UI's
relative-time formatter instead of a hand-rolled minutes count.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): pin a confirmed stale-lock reclaim to the record the user approved

--yes reclaimed whatever record held the lock at that moment, so a lock a
same-named machine wrote while the dialog sat open could be cleared.
orcad now takes --expect-nonce and refuses unless the lock still holds
it; the dry run reports the nonce, the desktop sends it back on confirm,
and the managed slot gate requires a build that knows the pin. A
reclaimed result carries the evidence it acted on.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(settings): keep the Recover server dialog open through its own reclaim

The dialog's mount was gated on the status that offered it, and the
reclaim clears that note before its answer arrives, so the result was
never shown and the card kept the dialog marked open. Only the triggers
are gated now; an open dialog keeps the server it opened for until the
user closes it.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(settings): put the Recover server confirmation on the security surface

It asks permission to run a remote command that clears an ownership lock
and starts server code, which STYLEGUIDE puts on that surface.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): trust a gone launch PID only on the machine that launched it

The PID file sits in a home other login nodes may share, so a wait exec
that lands on another node read a starting server as exited. Both launch
paths now record the launching machine's boot id and node name beside
the PID, and the readiness wait ends early only when its own exec reads
the same record. No record, a host without a boot id, or another machine
keeps the deadline. The PID path is named once in the launch command so
it stays within its sent-byte ceiling.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): keep a slot's unreadable stale-lock answer from offering a manual delete

A slot that passed the gate but gave no JSON, or a status this client
does not know, read as unsupported and offered rm of the lock. It now
reads as no-answer, which points to the slot's own reclaim command, and
unsupported is left to a slot without the command, a non-Linux host or
no active version.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* style(settings): use the text-2xs step for the stale-lock command block

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(mobile): offer the desktop-recovery hint on the same refusal the desktop does

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* test(orcad): name the extra fields a fake stale-lock answer carries

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(settings): end the Recover server flow once the lock is cleared, and keep an unpinned reclaim manual

After a reclaim whose restart failed, the dialog kept the old evidence,
the enabled confirmation and manual recovery for a lock that was gone.
It now shows only that the lock was cleared and why the server did not
start, with Close. A slot that has the reclaim but not the nonce pin is
now told apart from one without the reclaim, and is pointed to its own
reclaim command rather than rm.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): keep a stale-lock reclaim's result when the restart after it throws

A wake or tunnel rebuild that threw after the lock was cleared discarded
the reclaimed result, so the caller skipped its reconnect and note
clearing and the dialog kept a confirmation for a lock that was gone.
The restart's failure now becomes the result's notRestarted reason.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* fix(orcad): report a reclaimed server as started when only its forward failed to follow

A tunnel rebuild that failed after the wake started the server was
reported as the server not starting. The rebuild now goes through
followLaunchedServerPort, as other relaunches do, which warns and leaves
the forward to the next connect. Also drops a caller test that only
covered behaviour the fix did not change.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

* test(mobile): re-record the execution-target goldens for the recovery hint field

The SSH gate's projected state gained `needsDesktopServerRecovery`, which
is false for every recorded reply (no recorded host sends a managed
server refusal). Re-recorded with `pnpm --dir mobile rpc:record` for the
two pilot recordings and the four execution-target reply partitions; the
only change is that field, plus the value pool reordering around it.

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>

---------

Co-authored-by: Jesse Sanford <108698+jessesanford@users.noreply.github.com>
2026-10-23 17:57:22 -07:00

222 lines
7.5 KiB
TypeScript

import type { PublicKnownRuntimeEnvironment } from './runtime-environments'
import type { OrcadTerminalCensus } from './orcad-terminal-census'
import type { OrcadMigrationBlocker } from './orcad-migration/orcad-migration-preflight'
import type { OrcadMigrationSourceCutoverPhase } from './orcad-migration/orcad-migration-source-cutover'
import type { OrcadDaemonRetirementVerdict } from './orcad-stop-request'
import type { SshManagedServerLinkedProject } from './ssh-types'
import type { OrcadStaleLockEvidence } from './orcad-stale-lock-reclaim'
export const ORCAD_MANAGED_REMOTE_PORT = 6_768
/** Refused because terminals are, or may be, still running on the host. */
export type OrcadManagedRefusal = {
outcome: 'refused'
verdict: 'live' | 'unverifiable'
code: string
reason: string
/** Only saved state blocks, which a confirmed "Move anyway" leaves on this desktop. */
canLeaveBehind?: true
}
export type OrcadManagedDeferral = {
outcome: 'deferred'
candidateVersion: string
code: string
reason: string
/** False when force cannot make the rejected lifecycle transition safe. */
forceable?: boolean
}
export type OrcadManagedDeployResult =
| {
outcome: 'created' | 'updated' | 'already-current'
environment: PublicKnownRuntimeEnvironment
activeVersion: string
}
| OrcadManagedDeferral
export type OrcadManagedRollbackResult =
| {
outcome: 'rolled-back'
environment: PublicKnownRuntimeEnvironment
activeVersion: string
discarded: string[]
}
| { outcome: 'refused' | 'failed'; code: string; reason: string }
/** A recovery that needs the operator to accept restoring a snapshot over changed state. */
export const ORCAD_RECOVERY_CHANGED_STATE_CODE = 'orcad_recovery_changed_state'
export type OrcadManagedRecoveryResult =
| { outcome: 'none' }
| { outcome: 'pending'; code: string; reason: string }
| {
outcome: 'recovered'
resolution: 'committed' | 'restored-incumbent'
activeVersion: string | null
environment: PublicKnownRuntimeEnvironment
}
| OrcadManagedRefusal
/** Only a proven `exited` unlinks the server locally; anything less keeps it linked. */
export type OrcadManagedStopResult =
| {
outcome: 'unlinked'
verdict: 'exited'
environmentId: string
sshTargetId: string
stoppedVersion: string | null
retirement: OrcadDaemonRetirementVerdict | null
}
| OrcadManagedRefusal
/**
* Restarting the server from the current SSH session. Refused while any terminal runs or can't be
* counted (`liveSessions: null`): the terminal daemon outlives orcad and keeps what it inherited.
*/
export type OrcadManagedRestartResult =
| { outcome: 'restarted'; environmentId: string }
| { outcome: 'terminals-running'; liveSessions: number | null }
| { outcome: 'not-restarted'; reason: string }
/** A dry run, or a confirmation pinned to the lock record that dry run showed. */
export type OrcadManagedStaleLockReclaimRequest =
| { yes: false }
| { yes: true; expectNonce: string }
/** `orcad --reclaim-stale-lock` run on a managed host's active slot (#32177). */
export type OrcadManagedStaleLockReclaimResult =
| {
status: 'none' | 'eligible' | 'refused'
message: string
lockPath: string
evidence?: OrcadStaleLockEvidence
}
/** `notRestarted` says why the server did not start again once its lock was cleared. */
| {
status: 'reclaimed'
environmentId: string
message: string
lockPath: string
/** The record the reclaim acted on. */
evidence?: OrcadStaleLockEvidence
notRestarted?: string
}
/**
* The host can't run the command from here; nothing changed. `reclaimWithoutPin`: its orcad has
* the reclaim, but not the pin a confirmation from here needs, so the user runs the reclaim
* themselves. Otherwise `lockPath` names the lock to delete by hand.
*/
| { status: 'unsupported'; message: string; lockPath?: string; reclaimWithoutPin?: true }
/** The slot runs the command but gave no answer this client understands; nothing is known. */
| { status: 'no-answer'; message: string }
/** Another lifecycle step holds the host; nothing changed. */
| { status: 'busy'; message: string }
/**
* Unlinks the server from this desktop without reaching its host, for a host that is gone or
* unreachable. Never a claim of exit: the server may still run there, so the verdict stays
* `unverifiable`.
*/
export type OrcadManagedForgetResult =
| {
outcome: 'forgotten'
verdict: 'unverifiable'
environmentId: string
sshTargetId: string
}
| OrcadManagedRefusal
export type OrcadManagedCancelStopResult =
| { outcome: 'none' }
/** The stop was withdrawn before orcad acted on it; the server keeps serving. */
| { outcome: 'canceled'; activeVersion: string }
/** orcad had already exited; finish with stop to unlink the server. */
| { outcome: 'already-stopped' }
| OrcadManagedRefusal
export type OrcadManagedRuntimeStatus = {
environmentId: string
sshTargetId: string
activeVersion: string | null
previousVersion: string | null
activatedAt: string | null
rollbackAvailable: boolean
recovery:
| {
operation: 'activate'
phase: 'prepared' | 'incumbent-stopped' | 'snapshot-captured' | 'candidate-ready'
version: string
startedAt: string
}
| {
operation: 'rollback'
phase:
| 'prepared'
| 'incumbent-stopped'
| 'rescue-captured'
| 'rollback-state-restored'
| 'target-ready'
version: string
startedAt: string
}
| {
operation: 'decommission'
phase: 'prepared' | 'stop-dispatched' | 'process-exited'
version: string
startedAt: string
}
| null
/** Terminals the daemon runs; `null` counts are unverifiable and block updates and stops. */
terminals: OrcadTerminalCensus
/** An unfinished dormant migration into this server; a rollback is refused while it runs. */
migration: {
migrationId: string
phase: OrcadMigrationSourceCutoverPhase
startedAt: string
} | null
/** The last update this client deferred for this server, cleared once one goes through. */
deferredUpdate: (OrcadManagedDeferral & { deferredAt: string }) | null
}
export type OrcadManagedConversionResult =
| {
/** Committed on the server and retired from the SSH host. */
outcome: 'converted'
environment: PublicKnownRuntimeEnvironment
migrationId: string
/** Projects the server already held, so this desktop's own state for them did not move. */
linkedProjects?: SshManagedServerLinkedProject[]
}
| OrcadManagedDeferral
| OrcadManagedRefusal
export type OrcadManagedPendingMigrationRow = {
migrationId: string
environmentId: string
name: string
sshTargetId: string
phase: OrcadMigrationSourceCutoverPhase
startedAt: string
}
export type OrcadDeltaMoveRow = {
kind: 'repository' | 'folder-workspace' | 'project-group'
id: string
label: string
}
/** What moving a host's newer projects would add, and what the server will keep as it is. */
export type OrcadDeltaMovePreview = {
sshTargetId: string
environmentId: string
added: OrcadDeltaMoveRow[]
/** Projects an older build changed or removed; the server keeps its own copy of these. */
notReflected: { edited: OrcadDeltaMoveRow[]; removed: OrcadDeltaMoveRow[] }
blockers: OrcadMigrationBlocker[]
}
export type OrcadDeltaMoveResult =
| { outcome: 'moved'; migrationId: string }
| { outcome: 'refused'; code: string; reason: string; blockers?: OrcadMigrationBlocker[] }