mirror of
https://github.com/stablyai/orca.git
synced 2026-09-25 08:02:31 +00:00
* fix(daemon): escape the terminal daemon into its own systemd scope so a service restart no longer kills every live PTY Root cause: daemon-launched-child.ts forks the detached terminal daemon with detached: true, which escapes the POSIX process group (setsid) but never the systemd cgroup. Every PTY the daemon owns is itself an undetached direct child of the daemon (native-pty-spawn.ts). Under a combined systemd unit (Type=simple, KillMode=mixed, per docs/reference/headless-linux-server.md), a systemctl restart/stop SIGKILLs every process still in the cgroup at the stop timeout -- the daemon and every live terminal -- even though the codebase already has a fully-built adoption/reattachment path for a surviving daemon (orcad-entry.ts's refreshRestoredOrchestrationAuthority + reconcileLegacyWorkerTerminals, gated on daemonOwnsFreshPersistentPtys()). That path never fires today because the daemon never survives long enough. Fix: when systemd is actually supervising the process and the OS user has a reachable systemd --user manager (isDurableDaemonScopeSupported(), Linux only), launch the daemon via systemd-run --user --scope so it lands in a cgroup that is a sibling of the service unit's cgroup, not a descendant of it. A systemctl restart of the combined unit then never reaches it. Any failure of the scoped launch (no reachable bus, D-Bus policy rejection, etc.) falls back transparently to the existing plain fork() launch, so every platform/environment without this capability is unaffected. The daemon self-detects its own resulting cgroup scope via /proc/self/cgroup (detectOwnCgroupScopeUnit()) rather than trusting the launcher's intent, and publishes it as cgroupUnit in its pid record and orcad's health/readiness payload (health.terminalDaemon.cgroupUnit), so a running deployment can be observed to confirm the fix actually engaged. No new session registry is added: the existing daemon pid-record + adoption protocol (publishDaemonPidFile, daemon-pid-record-quarantine.ts's dead-record reclaim, refreshRestoredOrchestrationAuthority) already implements durable, crash-safe reattachment for a surviving daemon -- it was simply never exercised against a full unit restart before now. Proven via a systemd-in-Docker recovery test: a live PTY session's shell process, its daemon, and the daemon's cgroup scope were all confirmed unchanged across a real systemctl restart of a Type=simple/KillMode=mixed unit, while the main process pid changed (confirming the unit actually restarted) and the new process's health payload recognized the surviving daemon as adopted and live. A fresh write into the same PTY post-restart reached the same running shell. Ordinary terminal create/work/release and the #18789/#18790 worker-release reap-fix regression tests are unaffected. Fixes stablyai/orca#19408 * fix(daemon): probe the real per-UID XDG_RUNTIME_DIR before trusting the process's own env isDurableDaemonScopeSupported()/buildDurableDaemonScopeCommand() trusted the current process's own XDG_RUNTIME_DIR env var first, falling back to /run/user/<uid> only when that var was unset entirely. On mtl-02, orca-serve@factory.service's RuntimeDirectory= hardening directive makes systemd export XDG_RUNTIME_DIR=/run/orca_serve/factory into the unit's process -- a private scratch dir that shares the env var's name but has nothing to do with the user session bus. /proc/<pid>/environ on that host confirmed exactly that path plus DBUS_SESSION_BUS_ADDRESS=disabled:, while the real bus was reachable the whole time at /run/user/985 (confirmed via systemctl --user is-system-running with that dir exported by hand). The probe treated the hardened override as authoritative, found no bus socket there, and reported unsupported on every launch -- so the cgroup-escape fix from #19408/#19430 never actually engaged on real hardware, even though tonight's factory deployment picked it up. Fix: resolveUserRuntimeDir() now always tries the conventional /run/user/<uid> path first (computed independently via getuid(), never trusted from env), checking for a genuinely connectable bus socket via statSync(...).isSocket() rather than a bare existsSync. It falls back to the process's own XDG_RUNTIME_DIR only when that canonical path has no reachable bus -- covering hosts that legitimately have no /run/user/<uid> at all but do have a working bus wherever their own environment points. buildDurableDaemonScopeCommand() now explicitly sets XDG_RUNTIME_DIR to whichever path this resolution picked, rather than inheriting the spread env's (possibly hardened-wrong) value. Both isDurableDaemonScopeSupported() and buildDurableDaemonScopeCommand() gained an injectable canonicalRuntimeDir parameter (defaulting to the real computed path) so tests can exercise the hardened-override scenario deterministically with a real, connectable AF_UNIX socket fixture instead of the live host's actual runtime directory. Docker's stock jrei/systemd-ubuntu test container never had this hardening directive, so this gap was structurally invisible to the container-based verification in #19430 -- only caught against real mtl-02 hardware. * fix(daemon): report the daemon's own pid over the ready handshake, not systemd-run's The launcher used to infer the daemon's identity pid from the immediate spawned child (`child.pid`). On the durable-scope path that child is `systemd-run --user --scope`, not the daemon, so the launcher was asserting an identity it had no authority over. `DaemonReadyIdentity` now carries a required `pid` populated from `process.pid` inside the daemon itself, and `daemon-launched-child.ts` takes `launchedIdentity.pid` from that self-report. Both sides of the `holdDaemonAdoptionLease` pid comparison therefore originate inside the daemon process, which is the idiom this branch already uses for cgroup membership (`detectOwnCgroupScopeUnit` reads `/proc/self/cgroup` rather than trusting what the launcher intended). Note on the reported consequence: `systemd-run --scope` registers its *own* pid on the transient scope unit and then `execvpe()`s the target command -- same pid, no intermediate process -- so adoption did not in fact fail on systemd >= 206 (verified against systemd 255.4-1ubuntu8.17 and current main, `src/run/run.c` `start_transient_scope()`). The fix stands on its own merits: it removes a silent dependency on that exec-vs-fork implementation detail, which a `systemd-run` shim earlier in PATH or any future systemd change would have broken with no diagnostic. `terminateLaunchedDaemonChild` was audited and deliberately left on `child.pid`: for the same execve-preserves-pid reason that pid is either still systemd-run mid-scope-setup (killing it correctly aborts the launch) or already the daemon, so it targets the right process either way. Regression coverage: `daemon-launched-child-identity.test.ts` pins the identity source, and `daemon-ready-identity.test.ts` gains pid-validation cases. Ready-message fixtures across the `daemon-init-*` suites were updated for the now-mandatory field. Addresses: https://github.com/stablyai/orca/pull/19430#discussion_r3953722704 https://github.com/stablyai/orca/pull/19430#discussion_r3954346518 * test(daemon): assert cgroupUnit in the pid-file parse contract `parseDaemonPidFile` returns `cgroupUnit` on every branch as of the durable-scope commit on this branch, but five exhaustive `toEqual` assertions in daemon-health.test.ts still described the pre-scope shape, so they failed on the branch independently of any later change. Adds the field to those expectations. Deliberately not relaxed to `toMatchObject`: asserting the full parsed shape is what makes these tests catch a field silently dropped from the pid-file contract. * refactor(daemon): resolve the canonical user runtime dir at one point The per-UID path cannot change for a live process, so compute it once into a module const instead of threading the same default call through three signatures, and drop the try/catch around a getuid() that cannot throw once it exists. Trims the module prose to the non-obvious facts and corrects the pid-file record comment: an unscoped daemon writes null; only records no daemon wrote are absent. * test(daemon): clean up the cgroup-scope fixtures and assert a verdict The cgroup fixture tracked only the file it wrote, leaking one temp dir per case. Drains both fixture lists with splice so the pop-may-be-undefined guards go away, and replaces a not-throw/typeof-boolean pair with the verdict it was circling: no resolvable runtime dir means unsupported. * refactor(daemon): share the detached child options across both launch paths cwd, detached and stdio were repeated in the fork and systemd-run branches, which left the two comments explaining them hovering over the env block instead. Names them once so each branch carries only its own delta. * refactor(daemon): validate the ready pid like every other field typeof-first narrows the value, so the two 'as number' casts the isSafeInteger check needed disappear and the pid guard reads like the startedAtMs guard below it. * fix(daemon): don't retry the launch unscoped after losing the endpoint race A scoped attempt that lost the endpoint to another daemon was retried unscoped: a second doomed fork, a misleading 'cgroup-scope launch failed' warning, and the same DaemonEndpointUnavailableError the caller was already going to adopt on. Rethrows it instead, since no launch mode can win a race that is already lost. Also drops a private alias for DaemonChildSpawnOptions and the two 'as number' casts on child.pid in the startup-failure cleanup. * fix(daemon): unlink the pid record by the pid the daemon published The record holds the daemon's self-reported pid, so match on that rather than on the immediate child's, which is the systemd-run wrapper's until it execs. * fix(daemon): route the scope launch through the child-process chokepoint The two files this PR added imported `node:child_process` directly, which `child-process-import-boundary.test.ts` fails on deterministically: the offender count went 155 -> 157 against a pin of exactly 155. Raising the pin or listing the files is what that test explicitly forbids, and the allowlist's own note says a split "moved the import, it did not add one" -- so the fix is to get both new files off the module and put the count back at 155. - `daemon-cgroup-scope.ts`: the `systemd-run --version` probe now uses `runProcessSync` instead of `execFileSync`, so it gets the shared spawn decisions. Kept synchronous deliberately: `launchDaemonChild` attaches the readiness listener in the same tick it is called, and an await before the spawn moves the child past that tick. A non-zero exit is data rather than a throw here, so the verdict now checks `code === 0 && !timedOut`. - `daemon-launched-child-spawn.ts`: the scoped launch uses `spawnProcess`, and the long-standing unscoped launch keeps `fork` semantics through a new `forkProcess`. - `src/shared/child-process/fork-process.ts`: the fork arm of the chokepoint. `spawnProcess` cannot express a Node child with an IPC channel started from a module path under an overridden `execPath`, and the existing launch tests are written against `fork`'s contract, so a spawn rewrite would have changed module resolution, `execPath` and `execArgv` at once. It passes `windowsHide: true` -- the flag every other call site in that directory sets, reachable via an assertion because `ForkOptions` omits it -- which keeps `windows-console-visibility.test.ts` at its pin of 65 too. Both ratchets pass with both pins and both allowlists untouched. Docs: `orcad-operations.md` and `headless-linux-server.md` still described the limitation this PR removes as permanent. Both now describe the durable-scope survival path and its preconditions (systemd as PID 1, a reachable user bus / `loginctl enable-linger`, `systemd-run` on PATH), and scope the old text to the unscoped-fallback case, pointing at `health.terminalDaemon.cgroupUnit` as the way to tell the two apart on a running host. * fix(daemon): seal the cgroup capability probe from the host and correct KillMode=mixed docs The capability probe consulted the host's own /run/systemd/system marker and spawned the real systemd-run binary, so the hermetic unit tests could only pass on a systemd host (and fail closed otherwise, even with faked bus sockets). - Thread systemdBootPath and runVersionProbe as test seams through isDurableDaemonScopeSupported, defaulting to the real boot marker and systemd-run --version probe in production. - Narrow the injected probe to the ProcessResult slice it consumes. - Cover: no-systemd-boot, non-zero probe exit, and probe-timeout cases. - Correct KillMode=mixed semantics in the docs: the cgroup-wide SIGKILL fires the instant the main process exits, not after TimeoutStopSec; document the Docker-container caveat and add KillMode=mixed to the multi-service template. * fix(daemon): satisfy assertion checks in scoped launch * fix(daemon): satisfy anti-slop and console guards * test(serve): update shutdown docs assertions for daemon scope * fix(daemon): migrate adopted legacy scopes * docs: qualify restart safety by daemon scope * docs(daemon): qualify Upgrade restart prose with durable scope caveat Align the Upgrade section in docs/reference/headless-linux-server.md with the earlier preservation section and docs/reference/orcad-operations.md: a service restart terminates live processes only when running under the unscoped fallback, and stops should be treated as destructive unless health.terminalDaemon.cgroupUnit names an orca-daemon-*.scope. Update the shutdown workflow test assertion in config/scripts/headless-serve-shutdown-workflow.test.mjs to match. * fix(daemon): harden legacy scope migration --------- Co-authored-by: Lesley Murfin <260182349+LesleyMurfin@users.noreply.github.com> Co-authored-by: m4air <m4air@Mac.localdomain>
201 lines
9.9 KiB
JavaScript
201 lines
9.9 KiB
JavaScript
import { readFileSync } from 'node:fs'
|
|
|
|
import { parse } from 'yaml'
|
|
import { describe, expect, it } from 'vitest'
|
|
|
|
const workflow = parse(readFileSync('.github/workflows/pr.yml', 'utf8'))
|
|
const headlessLinuxGuide = readFileSync('docs/reference/headless-linux-server.md', 'utf8')
|
|
const signalCase = readFileSync('config/docker/headless-serve-shutdown/run-signal-case.sh', 'utf8')
|
|
const shutdownDockerRunner = readFileSync(
|
|
'config/scripts/run-headless-serve-shutdown-docker.mjs',
|
|
'utf8'
|
|
)
|
|
const shutdownDockerfile = readFileSync('config/docker/headless-serve-shutdown/Dockerfile', 'utf8')
|
|
const desktopStartupOracle = readFileSync(
|
|
'config/docker/headless-serve-shutdown/run-appimage-desktop-startup-case.sh',
|
|
'utf8'
|
|
)
|
|
const headlessLinuxProse = headlessLinuxGuide.replace(/\s+/g, ' ')
|
|
|
|
function readSystemdUnitBlocks(doc, unitName) {
|
|
const escapedUnitName = unitName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
|
|
return [...doc.matchAll(new RegExp(`^# /etc/systemd/system/${escapedUnitName}$`, 'gm'))].map(
|
|
(match) => {
|
|
const start = match.index + match[0].length
|
|
const end = doc.indexOf('```', start)
|
|
const nextUnitHeaderOffset = doc.slice(start).search(/^# \/etc\/systemd\/system\/.+$/m)
|
|
const nextUnitHeader = nextUnitHeaderOffset === -1 ? -1 : start + nextUnitHeaderOffset
|
|
if (end === -1 || (nextUnitHeader !== -1 && end > nextUnitHeader)) {
|
|
throw new Error(`Missing closing code fence for ${unitName}`)
|
|
}
|
|
return doc.slice(start, end)
|
|
}
|
|
)
|
|
}
|
|
|
|
describe('headless serve shutdown PR gate', () => {
|
|
it('reads only exact, closed systemd unit blocks', () => {
|
|
expect(
|
|
readSystemdUnitBlocks('# /etc/systemd/system/orca-serveXservice\n```', 'orca-serve.service')
|
|
).toEqual([])
|
|
expect(() =>
|
|
readSystemdUnitBlocks('# /etc/systemd/system/orca-serve.service\n', 'orca-serve.service')
|
|
).toThrow('Missing closing code fence for orca-serve.service')
|
|
expect(() =>
|
|
readSystemdUnitBlocks(
|
|
'# /etc/systemd/system/orca-serve.service\n' +
|
|
'KillMode=mixed\n' +
|
|
'# /etc/systemd/system/other.service\n```',
|
|
'orca-serve.service'
|
|
)
|
|
).toThrow('Missing closing code fence for orca-serve.service')
|
|
})
|
|
|
|
it('packages Linux artifacts before running the Docker signal oracle', () => {
|
|
const steps = workflow.jobs.package.steps
|
|
const packageStep = steps.find((step) => step.name === 'Package unpacked app')
|
|
const markerStep = steps.find((step) => step.name === 'Verify root-package marker payloads')
|
|
const shutdownStep = steps.find((step) => step.name === 'Verify headless serve signal shutdown')
|
|
|
|
expect(workflow.jobs.package['timeout-minutes']).toBe(90)
|
|
expect(packageStep.run).toContain('--linux AppImage deb rpm --x64 --publish never')
|
|
expect(markerStep.run).toContain('dpkg-deb --fsys-tarfile')
|
|
expect(markerStep.run).toContain('rpm2cpio')
|
|
expect(steps.indexOf(markerStep)).toBeGreaterThan(steps.indexOf(packageStep))
|
|
expect(shutdownStep.run).toBe(
|
|
'node config/scripts/run-headless-serve-shutdown-docker.mjs --appimage dist/orca-linux.AppImage --all-entrypoints'
|
|
)
|
|
expect(steps.indexOf(shutdownStep)).toBeGreaterThan(steps.indexOf(packageStep))
|
|
expect(steps.indexOf(shutdownStep)).toBeGreaterThan(steps.indexOf(markerStep))
|
|
expect(
|
|
steps.filter((step) => step.run?.includes('run-headless-serve-shutdown-docker.mjs'))
|
|
).toHaveLength(1)
|
|
})
|
|
|
|
it('keeps readiness polling finite and leak-free', () => {
|
|
expect(signalCase).toContain('read_ready_line()')
|
|
expect(signalCase).toContain("sed -u -n 's/^[^{]*//p'")
|
|
expect(signalCase).toContain('startup_timeout_seconds=${ORCA_STARTUP_TIMEOUT_SECONDS:-180}')
|
|
expect(signalCase).toContain('startup_deadline=$((SECONDS + startup_timeout_seconds))')
|
|
expect(signalCase).toContain('while (( SECONDS < startup_deadline )); do')
|
|
expect(signalCase).toContain('kill -0 "$app_pid" 2>/dev/null || break')
|
|
expect(signalCase).toContain(
|
|
"jq's `inputs` waits for EOF even when wrapped in `first`, so a tail -F"
|
|
)
|
|
expect(signalCase).not.toContain('tail --pid=')
|
|
})
|
|
|
|
it('gives owned shutdown state a bounded cleanup grace', () => {
|
|
expect(signalCase).toContain('for shutdown_poll in {0..50}; do')
|
|
expect(signalCase).toContain('[[ -z "$listener_after" && -z "$owned_residue" ]]')
|
|
expect(signalCase).toContain('((${#survivors[@]} == 0))')
|
|
expect(signalCase).toContain('((shutdown_poll < 50)) && sleep 0.1')
|
|
})
|
|
|
|
it('checks that a serving-electron signal target owns the ready socket', () => {
|
|
const ssRecord =
|
|
'LISTEN 0 128 127.0.0.1:41235 0.0.0.0:* users:(("orca-ide",pid=23,fd=7),("orca-ide",pid=25,fd=8))'
|
|
expect([...ssRecord.matchAll(/pid=([0-9]+)/g)].map((match) => match[1])).toEqual(['23', '25'])
|
|
expect(signalCase).toContain(
|
|
'listener_before_pids=$(grep -oE \'pid=[0-9]+\' <<<"$listener_before" | cut -d= -f2 || true)'
|
|
)
|
|
expect(signalCase).toContain('signal_target_pid=$(head -n1 <<<"$listener_before_pids")')
|
|
expect(signalCase).toContain('outside the entrypoint process tree')
|
|
})
|
|
|
|
it('runs the original AppImage desktop startup oracle before extraction and signals', () => {
|
|
expect(shutdownDockerfile).toContain(
|
|
'COPY run-appimage-desktop-startup-case.sh /usr/local/bin/run-appimage-desktop-startup-case'
|
|
)
|
|
const startupCall = shutdownDockerRunner.indexOf(
|
|
'runDesktopStartupOracle({ image, appImage, platform })'
|
|
)
|
|
const extractionCall = shutdownDockerRunner.indexOf(
|
|
"'timeout --kill-after=10s 120s /input/orca.AppImage --appimage-extract"
|
|
)
|
|
const signalLoop = shutdownDockerRunner.indexOf("for (const signal of ['INT', 'TERM'])")
|
|
expect(startupCall).toBeGreaterThan(-1)
|
|
expect(extractionCall).toBeGreaterThan(startupCall)
|
|
expect(signalLoop).toBeGreaterThan(startupCall)
|
|
expect(shutdownDockerRunner).toContain("'/usr/local/bin/run-appimage-desktop-startup-case'")
|
|
})
|
|
|
|
it('preserves startup logs when the launcher exits before its marker', () => {
|
|
expect(desktopStartupOracle).toContain('signal_process_group TERM || true')
|
|
expect(desktopStartupOracle).toContain('signal_process_group KILL || true')
|
|
expect(desktopStartupOracle).toContain('cat "$stdout_log" >&2 2>/dev/null || true')
|
|
expect(desktopStartupOracle).toContain('cat "$stderr_log" >&2 2>/dev/null || true')
|
|
expect(desktopStartupOracle).toContain(
|
|
'FAIL: desktop launcher exited before ${reason} (status=${observed_status})'
|
|
)
|
|
expect(desktopStartupOracle).toContain('ORCA_STARTUP_STATE_DIR_CLEANUP=1')
|
|
expect(desktopStartupOracle).toContain(
|
|
'[[ "$state_dir" =~ ^/tmp/orca-appimage-startup\\.[^/]+$ ]] || return 0'
|
|
)
|
|
})
|
|
|
|
it('requires the bound AppImage to be executable before launch and extraction', () => {
|
|
expect(desktopStartupOracle).toContain(
|
|
'[[ -x "$appimage" ]] || { echo "FAIL: AppImage is not executable: $appimage" >&2; exit 1; }'
|
|
)
|
|
expect(shutdownDockerRunner).toContain(
|
|
'\'test -r /input/orca.AppImage && test -x /input/orca.AppImage || { echo "FAIL: AppImage bind must be readable and executable" >&2; exit 1; }\''
|
|
)
|
|
})
|
|
|
|
it('gives the original AppImage enough bounded extraction space', () => {
|
|
expect(shutdownDockerRunner).toContain("'/tmp:rw,nosuid,nodev,exec,size=1g'")
|
|
})
|
|
|
|
it('keeps owned Xvfb alive during the documented systemd graceful stop', () => {
|
|
const serveUnits = readSystemdUnitBlocks(headlessLinuxGuide, 'orca-serve.service')
|
|
const ownedXvfbUnits = serveUnits.filter((unit) => !/^Environment=DISPLAY=/m.test(unit))
|
|
const managedXvfbUnits = serveUnits.filter((unit) => /^Environment=DISPLAY=/m.test(unit))
|
|
|
|
expect(ownedXvfbUnits).toHaveLength(1)
|
|
expect(ownedXvfbUnits[0]).toMatch(/^ExecStart=.*orca-linux\.AppImage serve.*$/m)
|
|
expect(ownedXvfbUnits[0]).toMatch(/^KillMode=mixed$/m)
|
|
expect(managedXvfbUnits).toHaveLength(1)
|
|
expect(managedXvfbUnits[0]).toMatch(/^KillMode=mixed$/m)
|
|
})
|
|
|
|
it('distinguishes persisted state from live work during a service restart', () => {
|
|
expect(headlessLinuxProse).toContain(
|
|
'The detached terminal daemon is preserved by a different mechanism: it is launched through `systemd-run --user --scope`'
|
|
)
|
|
expect(headlessLinuxProse).toContain(
|
|
'These guarantees preserve live processes only when the daemon is in its own'
|
|
)
|
|
expect(headlessLinuxProse).toContain(
|
|
'The unscoped fallback remains destructive: a service restart kills every terminal'
|
|
)
|
|
expect(headlessLinuxProse).toContain(
|
|
'Treat a stop as destructive unless `health.terminalDaemon.cgroupUnit` names an `orca-daemon-*.scope` on that host'
|
|
)
|
|
expect(headlessLinuxProse).toContain(
|
|
'A separately paired runtime is outside that boundary; local execution and SSH hosts reached through this runtime are not. An affected or unknown omission, missing scope, failed request or lost connection is `unverifiable`'
|
|
)
|
|
expect(headlessLinuxGuide).toContain(
|
|
'sudo -Hu orca /home/orca/.local/bin/orca-ide terminal list --json'
|
|
)
|
|
expect(headlessLinuxGuide).not.toContain('sudo -Hu orca orca-ide terminal list --json')
|
|
expect(headlessLinuxGuide).not.toContain('Two facts make this safe and predictable')
|
|
})
|
|
|
|
it('uses the registered CLI name from ordinary Linux shells', () => {
|
|
const commandRule =
|
|
'The registered Linux CLI command is `orca-ide`, not `orca`, to avoid shadowing the GNOME Orca screen reader.'
|
|
const substitutionRule =
|
|
"From an ordinary shell outside that service user's managed environment, substitute `orca-ide` for `orca` in commands below."
|
|
const censusCommand = '`sudo -Hu orca /home/orca/.local/bin/orca-ide terminal list --json`'
|
|
|
|
expect(headlessLinuxProse).toContain(commandRule)
|
|
expect(headlessLinuxProse).toContain(substitutionRule)
|
|
expect(headlessLinuxProse).toContain(censusCommand)
|
|
expect(headlessLinuxGuide).toContain('best-effort dispatcher at `$HOME/.local/bin/orca`')
|
|
expect(headlessLinuxProse.indexOf(substitutionRule)).toBeLessThan(
|
|
headlessLinuxProse.indexOf(censusCommand)
|
|
)
|
|
})
|
|
})
|