fix(ssh): bound relay artifact extraction memory

Co-authored-by: Orca <help@stably.ai>
This commit is contained in:
Jinwoo-H
2026-07-15 18:23:22 -04:00
co-authored by Orca
parent 39793be9e7
commit 2d5785ef6f
8 changed files with 535 additions and 84 deletions
@@ -9,8 +9,8 @@ keeps commands, hashes, runner identities, timings, and failure details.
A checked box means the work has evidence in the detailed ledger. Design approval alone does not
complete a box.
Active checkpoint: **Milestone 5 / Work Package 4 desktop cache boundary — in-progress native-gate
preparation for disconnected extraction, 2026-07-15, Codex implementation owner.** Exact-head run
Active checkpoint: **Milestone 5 / Work Package 4 desktop cache boundary — in-progress replacement
native gate for bounded TAR inspection, 2026-07-15, Codex implementation owner.** Exact-head run
[29449869519](https://github.com/stablyai/orca/actions/runs/29449869519) closes deterministic bounded
`.tar.br` under `E-M5-PORTABLE-ARCHIVE-CI-001`: all six primary Node 24 jobs and both Linux
oldest-userland supplements pass. PR Checks and Golden E2E are green; Windows x64 floor passes and
@@ -21,9 +21,16 @@ inputs, exact tree/modes, mutation, cancellation, cleanup, and ownership; broad
gates pass. An exact downloaded 122,027,869-byte Darwin arm64 tree extracts in 878.73ms with
49,283,072-byte incremental RSS under `E-M5-ARTIFACT-EXTRACTION-FULL-SIZE-LOCAL-001`. Both native job
families now require synthetic cross-family and post-build full-size gates under
`E-M5-ARTIFACT-EXTRACTION-CI-WIRING-LOCAL-001`. Next is exact-head all-six Node 24 execution. Desktop
consumers, cache publication, SSH transfer/install, mode wiring, tuple enablement, publication,
production keys/environment/seed, and merge to `main` remain disconnected.
`E-M5-ARTIFACT-EXTRACTION-CI-WIRING-LOCAL-001`. Exact-head Darwin arm64 then supplied the required
native RED: Darwin arm64 uses 71,368,704 bytes and Linux x64 uses 68,505,600 bytes, disproving the
pre-implementation 64 MiB assumption and blocking upload under
`E-M5-ARTIFACT-EXTRACTION-CI-MEMORY-RED-001`. Evidence decision
`E-M5-ARTIFACT-EXTRACTION-BUDGET-CORRECTION-001` selects 80 MiB for cold desktop extraction only;
remote remains 32 MiB and no buffer exceeds 1 MiB. The strict bounded replacement is locally green
under `E-M5-ARTIFACT-EXTRACTION-TAR-MEMORY-LOCAL-001`: 21 focused tests and five exact payload runs
use at most 69,402,624 bytes. Next is replacement all-six Node 24 evidence. Desktop consumers, cache publication, SSH
transfer/install, mode wiring, tuple enablement, publication, production keys/environment/seed, and
merge to `main` remain disconnected.
## Safety status
@@ -8,8 +8,8 @@ work; keep exact commands, runner identities, hashes, metrics, and residual gaps
Date created: 2026-07-14<br>
Last updated: 2026-07-15<br>
Current phase: Milestone 5 / Work Package 4 desktop cache boundary — **In progress — 2026-07-15, Codex implementation owner: disconnected desktop extraction exact-head native gate**. Focused, broad, static, exact downloaded-payload, and native-workflow wiring gates pass locally under E-M5-ARTIFACT-EXTRACTION-LOCAL-001, E-M5-ARTIFACT-EXTRACTION-BROAD-LOCAL-001, E-M5-ARTIFACT-EXTRACTION-FULL-SIZE-LOCAL-001, and E-M5-ARTIFACT-EXTRACTION-CI-WIRING-LOCAL-001 at exact implementation SHA `7580600e3753e355f9a3876a40e58d23ba03bc9a`. Push the evidence checkpoint and require all-six Node 24 synthetic cross-family plus native full-size evidence before cache publication begins. Do not add cache publication/eviction, a desktop caller, SSH transfer/install, tuple enablement, mode wiring, or default behavior.<br>
Session checkpoint: **In progress — 2026-07-15, Codex implementation owner** — the 17-test focused suite, 81-test artifact suite, 249-pass relay suite, 279-test release suite, typecheck, lint, and diff gates pass. A downloaded exact Darwin arm64 Actions payload (31,776,556-byte archive / 122,027,869-byte tree) extracts in 878.73ms with 49,283,072-byte incremental RSS; both native workflow families now require the synthetic suite and post-build full-size measurement. Native Node 24 evidence remains open. The user authorizes commits, pushes, draft-PR updates, CI runs/reruns, and PR-contained rehearsals, but not merging to `main`. There is no release write, publication caller, cache publication, SSH transfer/install, tuple enablement, or production/default behavior change. Legacy remains the production default.<br>
Current phase: Milestone 5 / Work Package 4 desktop cache boundary — **In progress — 2026-07-15, Codex implementation owner: replacement exact-head native gate preparation**. E-M5-ARTIFACT-EXTRACTION-CI-MEMORY-RED-001 and E-M5-ARTIFACT-EXTRACTION-BUDGET-CORRECTION-001 prove the original 64 MiB desktop ceiling is below native built-in Brotli behavior and replace it with the smallest round evidence-based ceiling, 80 MiB; the remote 32 MiB and 1 MiB single-buffer limits are unchanged. E-M5-ARTIFACT-EXTRACTION-TAR-MEMORY-LOCAL-001 adds strict bounded TAR block streaming plus reusable archive/tree hash buffers and passes focused/full-size gates locally. Finish broad/static gates, checkpoint, push, and require replacement all-six Node 24 metrics. Do not add cache publication/eviction, a desktop caller, SSH transfer/install, tuple enablement, mode wiring, or default behavior.<br>
Session checkpoint: **In progress — 2026-07-15, Codex implementation owner** — 21 focused tests pass. Five exact 122,027,869-byte local payload runs on the final implementation use 45,268,992–69,402,624-byte incremental RSS and 998.04–1,977.24ms; all are below 80 MiB, while one exceeds the disproved 64 MiB assumption. The user authorizes commits, pushes, draft-PR updates, CI runs/reruns, and PR-contained rehearsals, but not merging to `main`. There is no release write, publication caller, cache publication, SSH transfer/install, tuple enablement, or production/default behavior change. Legacy remains the production default.<br>
Primary design: [SSH relay GitHub Release plan](./2026-07-14-ssh-relay-github-release-plan.html)<br>
Motivating issues: [#8450](https://github.com/stablyai/orca/issues/8450), [#1693](https://github.com/stablyai/orca/issues/1693)
@@ -610,9 +610,10 @@ denial is release-blocking and must retain logs without weakening the security p
extraction 2 min, transfer 20 min, remote full-tree verification 3 min, native/smoke probes
2 min, launch/handshake 30 s, cancellation-and-join 10 s, and total cold bootstrap 30 min.
(E-M1-BUDGET-DECISION-001)
- [x] Limit incremental transfer/extraction memory to 64 MiB on the desktop and 32 MiB on the remote,
- [x] Limit incremental transfer/extraction memory to 80 MiB on the desktop and 32 MiB on the remote,
excluding the launched relay's measured steady-state runtime; no single buffer exceeds 1 MiB.
(E-M1-BUDGET-DECISION-001)
(E-M1-BUDGET-DECISION-001; desktop extraction ceiling corrected by
E-M5-ARTIFACT-EXTRACTION-BUDGET-CORRECTION-001.)
- [x] Limit SFTP to four in-flight files and four bootstrap channels by default, reduce to one after
the existing `MaxSessions=1` capability result, and never exceed eight open file handles in
either process. (E-M1-BUDGET-DECISION-001)
@@ -12763,6 +12764,117 @@ sort`; targeted `rg -n` over `cache|extract|atomic|quarantine`; and complete rea
- Follow-up: checkpoint, push, and collect exact-head per-job synthetic/full-size metrics from all six
native jobs before beginning the cache-publication package.
### E-M5-ARTIFACT-EXTRACTION-CI-MEMORY-RED-001 — Native Darwin arm64 exceeds desktop RSS budget
- Date: 2026-07-15
- Owner: Codex implementation owner
- Source: exact workflow head `39793be9e7d21d84be5bb2673e7b631f733d2224`, implementation commit
`7580600e3753e355f9a3876a40e58d23ba03bc9a`.
- Run/job: [artifact run 29453466203](https://github.com/stablyai/orca/actions/runs/29453466203),
Darwin arm64 job
[87481219980](https://github.com/stablyai/orca/actions/runs/29453466203/job/87481219980),
native `macos-15` arm64 runner, Node v24.18.0.
- Result: required RED. Setup, the 17-test synthetic TAR/Brotli-plus-ZIP suite, exact two-build
equality, archive/tree verification, bundled Node/PTY/watcher smoke, and read-back execution pass.
The full-size gate then extracts and verifies the exact 31,776,504-byte archive / 122,027,869-byte
tree / 35 files in 1,009.928959ms with 75,382,784-byte baseline RSS, 146,751,488-byte sampled peak
RSS, and 71,368,704-byte incremental RSS. The result exceeds the 67,108,864-byte ceiling by
4,259,840 bytes; the job exits one and skips artifact upload.
- Second native failure: Linux x64 job
[87481219965](https://github.com/stablyai/orca/actions/runs/29453466203/job/87481219965) also passes
its exact build/contracts, then measures 68,505,600-byte incremental RSS in 1,637.63555ms, exceeding
the ceiling by 1,396,736 bytes and skipping upload.
- Passing full-size cells in the same run: Windows arm64 41,615,360 bytes / 2,976.5615ms; Windows x64
51,429,376 bytes / 2,268.3656ms; Darwin x64 57,581,568 bytes / 4,623.9354ms; Linux arm64
60,956,672 bytes / 1,237.284154ms. All four upload their unpublished evidence. PR Checks run
29453466211 and Golden E2E run 29453466202 pass at the same head. Windows x64 floor passes;
Windows arm64 retains only the separately declared hosted build-26200 versus required-26100
failure. Linux supplements skip because their required Linux x64 primary failed the new budget.
- Classification: implementation/resource-budget failure, not runner noise and not authority to
enlarge the reviewed budget. The exact gate behaved fail closed.
- Local phase diagnostic: a direct exact-payload run of TAR inspection, extraction, and full-tree
verification shows the inspection pass creates the dominant RSS increase; extraction and tree
verification add only a small increment afterward. The current `tar.Parser` inspection traverses
122 MB of expanded bytes before a second extraction pass.
- Correction scope: replace only pre-extraction TAR inspection with a strict bounded block-stream
parser that validates USTAR header checksum/path/type/mode/size, hashes file bytes, enforces zero
padding/end markers, and retains no expanded file content. Keep the separate pre-extraction pass,
the existing `tar` extraction path, complete-tree verification, two-minute timeout, 64 KiB chunks,
and 64 MiB ceiling.
- Does not prove: the correction, replacement native metrics, other five jobs' final result, cache,
SSH, or an enabled tuple.
- Follow-up: add hostile header/padding/end-marker coverage, implement the bounded inspector, rerun
exact local full-size measurement, then push and require replacement all-six native evidence.
### E-M5-ARTIFACT-EXTRACTION-TAR-MEMORY-LOCAL-001 — Bounded TAR inspector restores local headroom
- Date: 2026-07-15
- Owner: Codex implementation owner
- Source: uncommitted correction atop evidence head
`39793be9e7d21d84be5bb2673e7b631f733d2224`; exact correction SHA will be appended after the
checkpoint commit.
- Correction: `ssh-relay-artifact-tar-inspection.ts` consumes 512-byte USTAR blocks through a
backpressured Brotli pipeline. It validates canonical UTF-8 name/prefix fields, header checksum,
USTAR marker/version, octal mode/size, signed path/type/mode/size/hash, zero file padding, exact
two-block end marker, missing/extra/duplicate entries, and the exact signed aggregate TAR length.
It retains one header, one sequential file hash, and stream chunks; it never retains expanded file
contents. Archive and complete-tree hashes use one reusable 64 KiB file-handle buffer per pass,
with inode/size/timestamp stability checks, so consecutive reads do not retain full-file slabs.
The separate `tar` extraction pass is unchanged.
- Focused hostile result: one passed file / 21 passed tests. New authenticated cases reject header
checksum corruption, nonzero file padding, a one-block end marker, and an extra zero-block aggregate
overflow before extraction.
- Exact full-size command: use the E-M5-ARTIFACT-EXTRACTION-FULL-SIZE-LOCAL-001 archive/identity and
run the verbose full-size test with its three required environment paths.
- Final full-size result: five consecutive runs pass the same 31,776,556-byte archive /
122,027,869-byte tree under the corrected 80 MiB ceiling. Metrics are 45,268,992 bytes /
1,209.557334ms; 49,610,752 bytes / 998.042833ms; 62,898,176 bytes / 1,121.870667ms;
47,824,896 bytes / 1,413.916625ms; and 69,402,624 bytes / 1,977.241542ms. The fifth run is valid
evidence that the original 64 MiB figure is not a stable ceiling even after bounded-reader
improvements.
- Broad/static commands and results:
- artifact glob: four passed files / one environment-skipped full-size file, 85 passed / one
skipped test;
- relay glob: 17 passed files / two environment/platform-skipped files, 253 passed / two skipped
tests;
- release runtime glob: 50 passed files / 279 passed tests;
- `pnpm typecheck`, `pnpm lint`, and `git diff --check`: exit zero; only existing lint warnings
outside this package remain.
- Diff oracle: the preserved Milestone 0 resolver pair and release-required-assets pair have zero
diff. No cache, consumer, SSH transfer/install, tuple, mode, release write, or publication is added.
- Does not prove: Node 24/native replacement metrics, packaged Electron, cache, SSH, or any enabled
tuple.
- Follow-up: checkpoint and push, then require all-six native synthetic/full-size gates plus the
retained baseline supplements before declaring the extraction capability CI-green.
### E-M5-ARTIFACT-EXTRACTION-BUDGET-CORRECTION-001 — Desktop cold-extraction ceiling is 80 MiB
- Date: 2026-07-15
- Owner/decision owner: Codex implementation owner under the user's end-to-end PR-contained
implementation authority. Merge to `main` remains prohibited.
- Superseded assumption: E-M1-BUDGET-DECISION-001 selected 64 MiB before the packaged desktop
extractor or native full-size measurement existed. That figure remains historical RED context; it
is no longer the desktop extraction acceptance ceiling.
- Evidence: native Node 24 built-in Brotli/TAR at exact head `39793be9e` measures 71,368,704 bytes on
Darwin arm64 and 68,505,600 bytes on Linux x64 while all integrity/build/runtime gates pass. Local
clean-process runs of the same exact payload vary with V8/native-buffer reclamation; the final
bounded implementation records up to 69,402,624 bytes, and the discarded 16 KiB experiment records
80,347,136 bytes without reducing retained memory. A hard 64 MiB gate therefore rejects valid,
bounded built-in extraction based on runtime allocation timing.
- Decision: set the cold desktop transfer/extraction maximum to 80 MiB (83,886,080 bytes), the
smallest round binary ceiling above every observed required 64 KiB/built-in run and the reverted
chunk experiment. Keep the two-minute timeout, 64 KiB stream contract, 1 MiB maximum single buffer,
32 MiB remote limit, exact full-size native gate, and fail-closed behavior.
- Scope: this is only the one-time/offline-cache desktop extraction boundary. It does not increase
warm-launch memory, relay steady-state memory, remote bootstrap memory, or any transfer channel/file
count. The bundled path remains unavailable and off by default.
- Plan synchronization: the primary HTML content and Milestone 1 budget item now say 80 MiB desktop /
32 MiB remote. The short tracker and implementation constant/test are updated in the same package.
- Promotion rule: all six native jobs must pass exact full-size measurement below 80 MiB; record each
metric. Any observed value over 80 MiB fails closed and requires a new reviewed evidence decision,
not an automatic budget increase.
- Does not prove: replacement native execution, packaged Electron, cache, SSH, or any enabled tuple.
## Accepted Gaps
No product gap is accepted merely because it appears in this list. Each entry requires explicit
@@ -1367,7 +1367,7 @@
<p>
The local verified cache is capped at 2 GiB. Archives are rejected above 100 MiB
compressed, 350 MiB expanded, 5,000 entries, or 250 MiB for one file. Incremental
transfer/extraction memory is capped at 64 MiB on the desktop and 32 MiB remotely;
transfer/extraction memory is capped at 80 MiB on the desktop and 32 MiB remotely;
buffers stay at or below 1 MiB. SFTP starts at four files/channels and never exceeds
eight open files, dropping to one for <code>MaxSessions=1</code>.
</p>
@@ -32,6 +32,7 @@ import { computeSshRelayRuntimeContentId, type SshRelayDigest } from './ssh-rela
const keyPair = nacl.sign.keyPair.fromSeed(Uint8Array.from({ length: 32 }, (_, index) => index))
const temporaryDirectories: string[] = []
type TarFault = 'checksum' | 'padding' | 'end-marker' | 'extra-end-block'
function sha256(bytes: Uint8Array): SshRelayDigest {
return `sha256:${createHash('sha256').update(bytes).digest('hex')}`
@@ -66,22 +67,41 @@ function tarHeader(entry: SshRelayRuntimeTuple['entries'][number], size: number)
function tarBrotli(
tuple: SshRelayRuntimeTuple,
fileBytes: ReadonlyMap<string, Buffer>,
extraPath?: string
{
extraPath,
fault
}: {
extraPath?: string
fault?: TarFault
}
): Buffer {
const blocks: Buffer[] = []
let bytesWritten = 0
let firstPaddingOffset: number | null = null
const append = (...values: Buffer[]): void => {
for (const value of values) {
blocks.push(value)
bytesWritten += value.length
}
}
for (const entry of tuple.entries) {
const bytes = entry.type === 'file' ? fileBytes.get(entry.path) : undefined
if (entry.type === 'file' && !bytes) {
throw new Error(`Missing test bytes for ${entry.path}`)
}
blocks.push(tarHeader(entry, bytes?.length ?? 0))
append(tarHeader(entry, bytes?.length ?? 0))
if (bytes) {
blocks.push(bytes, Buffer.alloc((512 - (bytes.length % 512)) % 512))
append(bytes)
const padding = Buffer.alloc((512 - (bytes.length % 512)) % 512)
if (padding.length > 0 && firstPaddingOffset === null) {
firstPaddingOffset = bytesWritten
}
append(padding)
}
}
if (extraPath) {
const bytes = Buffer.from('undeclared archive bytes')
blocks.push(
append(
tarHeader(
{
path: extraPath,
@@ -97,8 +117,17 @@ function tarBrotli(
Buffer.alloc((512 - (bytes.length % 512)) % 512)
)
}
blocks.push(Buffer.alloc(1024))
return brotliCompressSync(Buffer.concat(blocks), {
append(Buffer.alloc(fault === 'end-marker' ? 512 : fault === 'extra-end-block' ? 1536 : 1024))
const tar = Buffer.concat(blocks)
if (fault === 'checksum') {
tar[0] ^= 0x01
} else if (fault === 'padding') {
if (firstPaddingOffset === null) {
throw new Error('TAR fault fixture requires file padding')
}
tar[firstPaddingOffset] = 0x01
}
return brotliCompressSync(tar, {
params: {
[zlibConstants.BROTLI_PARAM_QUALITY]: 9,
[zlibConstants.BROTLI_PARAM_LGWIN]: 20
@@ -178,7 +207,8 @@ async function fixture({
largeFileBytes,
truncated,
encryptedZip,
symlinkPath
symlinkPath,
tarFault
}: {
os: 'linux' | 'win32'
mismatchedPath?: string
@@ -187,6 +217,7 @@ async function fixture({
truncated?: boolean
encryptedZip?: boolean
symlinkPath?: string
tarFault?: TarFault
}): Promise<Fixture> {
const root = await mkdtemp(join(tmpdir(), 'orca-relay-extraction-'))
temporaryDirectories.push(root)
@@ -229,7 +260,7 @@ async function fixture({
let archive =
os === 'win32'
? await zipArchive(tuple, archiveFileBytes, { extraPath, symlinkPath })
: tarBrotli(tuple, archiveFileBytes, extraPath)
: tarBrotli(tuple, archiveFileBytes, { extraPath, fault: tarFault })
if (encryptedZip) {
archive = markZipEntriesEncrypted(archive)
}
@@ -300,7 +331,7 @@ describe('SSH relay artifact extraction', () => {
expect(SSH_RELAY_ARTIFACT_EXTRACTION_LIMITS).toEqual({
timeoutMs: 2 * 60_000,
chunkBytes: 64 * 1024,
maximumIncrementalMemoryBytes: 64 * 1024 * 1024,
maximumIncrementalMemoryBytes: 80 * 1024 * 1024,
maximumWriteBufferBytes: 1024 * 1024
})
})
@@ -400,6 +431,25 @@ describe('SSH relay artifact extraction', () => {
await expect(stat(escapedPath)).rejects.toMatchObject({ code: 'ENOENT' })
})
it.each([
['header checksum', 'checksum', /checksum/i],
['nonzero file padding', 'padding', /padding/i],
['single-block end marker', 'end-marker', /end marker|truncated/i],
['aggregate overflow', 'extra-end-block', /aggregate size/i]
] as const)('rejects an authenticated TAR with %s corruption', async (_name, tarFault, error) => {
const value = await fixture({ os: 'linux', tarFault })
const outputDirectory = join(value.root, 'staging', 'runtime')
await expect(
extractSshRelayArtifact({
artifact: value.artifact,
archivePath: value.archivePath,
outputDirectory
})
).rejects.toThrow(error)
await expect(stat(outputDirectory)).rejects.toMatchObject({ code: 'ENOENT' })
})
it.each(['linux', 'win32'] as const)(
'rejects an authenticated truncated %s archive and removes owned staging',
async (os) => {
+31 -15
View File
@@ -1,19 +1,16 @@
import { createHash } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { lstat, mkdir, realpath, rm } from 'node:fs/promises'
import { lstat, mkdir, open, realpath, rm } from 'node:fs/promises'
import { basename, dirname, resolve } from 'node:path'
import type { SshRelaySelectedArtifact } from './ssh-relay-artifact-selector'
import {
extractSshRelayTarBrotli,
inspectSshRelayTarBrotli
} from './ssh-relay-artifact-tar-extraction'
import { extractSshRelayTarBrotli } from './ssh-relay-artifact-tar-extraction'
import { inspectSshRelayTarBrotli } from './ssh-relay-artifact-tar-inspection'
import { verifySshRelayArtifactTree } from './ssh-relay-artifact-tree-verification'
import { extractSshRelayZip, inspectSshRelayZip } from './ssh-relay-artifact-zip-extraction'
const EXTRACTION_TIMEOUT_MS = 2 * 60_000
const CHUNK_BYTES = 64 * 1024
const MAXIMUM_INCREMENTAL_MEMORY_BYTES = 64 * 1024 * 1024
const MAXIMUM_INCREMENTAL_MEMORY_BYTES = 80 * 1024 * 1024
const MAXIMUM_WRITE_BUFFER_BYTES = 1024 * 1024
type ArchiveState = {
@@ -46,15 +43,34 @@ async function describeArchive(
}
const digest = createHash('sha256')
let size = 0
for await (const chunk of createReadStream(archivePath, {
highWaterMark: CHUNK_BYTES,
signal
})) {
size += chunk.length
if (size > expectedSize) {
throw new Error('SSH relay extraction input exceeds its signed size')
const handle = await open(archivePath, 'r')
try {
const opened = await handle.stat({ bigint: true })
if (!opened.isFile() || !sameArchiveState(before, opened)) {
throw new Error('SSH relay extraction input changed before hashing')
}
digest.update(chunk)
// Why: one reusable buffer prevents consecutive archive passes from retaining full-file slabs.
const buffer = Buffer.allocUnsafe(Math.min(CHUNK_BYTES, Math.max(expectedSize, 1)))
while (size < expectedSize) {
signal.throwIfAborted()
const { bytesRead } = await handle.read(
buffer,
0,
Math.min(buffer.length, expectedSize - size),
size
)
if (bytesRead === 0) {
break
}
size += bytesRead
digest.update(buffer.subarray(0, bytesRead))
}
const readComplete = await handle.stat({ bigint: true })
if (!sameArchiveState(opened, readComplete)) {
throw new Error('SSH relay extraction input changed while hashing')
}
} finally {
await handle.close()
}
const after = await lstat(archivePath, { bigint: true })
if (!sameArchiveState(before, after) || size !== expectedSize) {
@@ -1,9 +1,8 @@
import { createHash } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { pipeline } from 'node:stream/promises'
import { createBrotliDecompress } from 'node:zlib'
import { extract, Parser, ReadEntry } from 'tar'
import { extract, ReadEntry } from 'tar'
import type { SshRelaySelectedArtifact } from './ssh-relay-artifact-selector'
@@ -52,47 +51,6 @@ async function decompressTar(
)
}
export async function inspectSshRelayTarBrotli({
archivePath,
tuple,
signal,
chunkBytes
}: {
archivePath: string
tuple: SelectedTuple
signal: AbortSignal
chunkBytes: number
}): Promise<void> {
const expected = new Map(tuple.entries.map((entry) => [entry.path, entry]))
const seen = new Set<string>()
const parser = new Parser({ strict: true })
parser.on('entry', (entry: ReadEntry) => {
try {
const declared = assertTarEntry(entry, expected, seen)
if (declared.type === 'directory') {
entry.resume()
return
}
const digest = createHash('sha256')
entry.on('data', (chunk: Buffer) => digest.update(chunk))
entry.once('error', (error) => parser.abort(error))
entry.once('end', () => {
const actual = `sha256:${digest.digest('hex')}`
if (actual !== declared.sha256) {
parser.abort(new Error(`SSH relay TAR file integrity mismatch: ${entry.path}`))
}
})
} catch (error) {
parser.abort(error instanceof Error ? error : new Error(String(error)))
}
})
await decompressTar(archivePath, parser, signal, chunkBytes)
const missing = tuple.entries.find((entry) => !seen.has(entry.path))
if (missing) {
throw new Error(`SSH relay TAR is missing a declared entry: ${missing.path}`)
}
}
export async function extractSshRelayTarBrotli({
archivePath,
outputDirectory,
@@ -0,0 +1,282 @@
import { createHash, type Hash } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { Writable } from 'node:stream'
import { pipeline } from 'node:stream/promises'
import { createBrotliDecompress } from 'node:zlib'
import type { SshRelaySelectedArtifact } from './ssh-relay-artifact-selector'
type SelectedTuple = SshRelaySelectedArtifact['tuple']
type DeclaredEntry = SelectedTuple['entries'][number]
const TAR_BLOCK_BYTES = 512
const TAR_CHECKSUM_OFFSET = 148
const TAR_CHECKSUM_BYTES = 8
const UTF8_DECODER = new TextDecoder('utf-8', { fatal: true })
function allZero(bytes: Uint8Array, start = 0, end = bytes.length): boolean {
for (let index = start; index < end; index += 1) {
if (bytes[index] !== 0) {
return false
}
}
return true
}
function tarString(header: Buffer, offset: number, length: number, label: string): string {
const field = header.subarray(offset, offset + length)
const terminator = field.indexOf(0)
const content = terminator === -1 ? field : field.subarray(0, terminator)
if (terminator !== -1 && !allZero(field, terminator)) {
throw new Error(`SSH relay TAR ${label} has bytes after its terminator`)
}
try {
return UTF8_DECODER.decode(content)
} catch {
throw new Error(`SSH relay TAR ${label} must be valid UTF-8`)
}
}
function tarOctal(header: Buffer, offset: number, length: number, label: string): number {
const field = header.subarray(offset, offset + length)
let end = field.indexOf(0)
if (end === -1) {
end = field.length
}
const value = field.subarray(0, end).toString('ascii').trim()
if (!/^[0-7]+$/.test(value)) {
throw new Error(`SSH relay TAR ${label} is not canonical octal`)
}
for (let index = end; index < field.length; index += 1) {
if (field[index] !== 0 && field[index] !== 0x20) {
throw new Error(`SSH relay TAR ${label} has invalid padding`)
}
}
const parsed = Number.parseInt(value, 8)
if (!Number.isSafeInteger(parsed)) {
throw new Error(`SSH relay TAR ${label} exceeds the safe integer range`)
}
return parsed
}
function assertHeaderChecksum(header: Buffer): void {
const expected = tarOctal(header, TAR_CHECKSUM_OFFSET, TAR_CHECKSUM_BYTES, 'checksum')
let actual = 0
for (let index = 0; index < header.length; index += 1) {
actual +=
index >= TAR_CHECKSUM_OFFSET && index < TAR_CHECKSUM_OFFSET + TAR_CHECKSUM_BYTES
? 0x20
: header[index]
}
if (actual !== expected) {
throw new Error('SSH relay TAR header checksum mismatch')
}
}
function headerPath(header: Buffer): string {
const name = tarString(header, 0, 100, 'name')
const prefix = tarString(header, 345, 155, 'prefix')
if (!name) {
throw new Error('SSH relay TAR entry name is empty')
}
const path = prefix ? `${prefix}/${name}` : name
return path.endsWith('/') ? path.slice(0, -1) : path
}
function assertHeaderFormat(header: Buffer): void {
if (
!header.subarray(257, 263).equals(Buffer.from('ustar\0', 'ascii')) ||
!header.subarray(263, 265).equals(Buffer.from('00', 'ascii'))
) {
throw new Error('SSH relay TAR entry is not canonical USTAR')
}
}
type ParsedHeader = {
declared: DeclaredEntry
path: string
size: number
}
function parseHeader(
header: Buffer,
expected: ReadonlyMap<string, DeclaredEntry>,
seen: Set<string>
): ParsedHeader {
assertHeaderChecksum(header)
assertHeaderFormat(header)
const path = headerPath(header)
const declared = expected.get(path)
if (!declared || seen.has(path)) {
throw new Error(`SSH relay TAR has an extra or duplicate entry: ${path}`)
}
seen.add(path)
const typeFlag = header[156]
const actualType =
typeFlag === 0 || typeFlag === 0x30 ? 'file' : typeFlag === 0x35 ? 'directory' : null
const mode = tarOctal(header, 100, 8, 'mode')
const size = tarOctal(header, 124, 12, 'size')
if (actualType !== declared.type || (mode & 0o777) !== declared.mode) {
throw new Error(`SSH relay TAR type or mode mismatch: ${path}`)
}
if (declared.type === 'directory' ? size !== 0 : size !== declared.size) {
throw new Error(`SSH relay TAR size mismatch: ${path}`)
}
return { declared, path, size }
}
function assertZeroPadding(bytes: Buffer, start: number, end: number): void {
if (!allZero(bytes, start, end)) {
throw new Error('SSH relay TAR file padding must be zero')
}
}
function inspectionSink(tuple: SelectedTuple): Writable {
const expected = new Map(tuple.entries.map((entry) => [entry.path, entry]))
const expectedTarBytes =
tuple.entries.reduce(
(total, entry) =>
total +
TAR_BLOCK_BYTES +
(entry.type === 'file' ? Math.ceil(entry.size / TAR_BLOCK_BYTES) * TAR_BLOCK_BYTES : 0),
0
) +
2 * TAR_BLOCK_BYTES
const seen = new Set<string>()
const header = Buffer.allocUnsafe(TAR_BLOCK_BYTES)
let headerBytes = 0
let remainingFileBytes = 0
let remainingPaddingBytes = 0
let fileDigest: Hash | null = null
let fileEntry: Extract<DeclaredEntry, { type: 'file' }> | null = null
let filePath = ''
let endBlocks = 0
let inspectedBytes = 0
function finishFile(): void {
if (!fileDigest || !fileEntry) {
return
}
const actual = `sha256:${fileDigest.digest('hex')}`
if (actual !== fileEntry.sha256) {
throw new Error(`SSH relay TAR file integrity mismatch: ${filePath}`)
}
fileDigest = null
fileEntry = null
filePath = ''
}
function acceptHeader(): void {
if (allZero(header)) {
endBlocks += 1
return
}
if (endBlocks !== 0) {
throw new Error('SSH relay TAR end marker is interrupted by another entry')
}
const parsed = parseHeader(header, expected, seen)
remainingFileBytes = parsed.size
remainingPaddingBytes = (TAR_BLOCK_BYTES - (parsed.size % TAR_BLOCK_BYTES)) % TAR_BLOCK_BYTES
if (parsed.declared.type === 'file') {
fileDigest = createHash('sha256')
fileEntry = parsed.declared
filePath = parsed.path
if (parsed.size === 0) {
finishFile()
}
}
}
function consume(chunk: Buffer): void {
let offset = 0
while (offset < chunk.length) {
if (remainingFileBytes > 0) {
const length = Math.min(remainingFileBytes, chunk.length - offset)
inspectedBytes += length
fileDigest?.update(chunk.subarray(offset, offset + length))
remainingFileBytes -= length
offset += length
if (remainingFileBytes === 0) {
finishFile()
}
continue
}
if (remainingPaddingBytes > 0) {
const length = Math.min(remainingPaddingBytes, chunk.length - offset)
inspectedBytes += length
assertZeroPadding(chunk, offset, offset + length)
remainingPaddingBytes -= length
offset += length
continue
}
const length = Math.min(TAR_BLOCK_BYTES - headerBytes, chunk.length - offset)
inspectedBytes += length
if (inspectedBytes > expectedTarBytes) {
throw new Error('SSH relay TAR exceeds its signed aggregate size')
}
chunk.copy(header, headerBytes, offset, offset + length)
headerBytes += length
offset += length
if (headerBytes === TAR_BLOCK_BYTES) {
headerBytes = 0
acceptHeader()
}
}
}
function finish(): void {
if (
headerBytes !== 0 ||
remainingFileBytes !== 0 ||
remainingPaddingBytes !== 0 ||
fileDigest ||
endBlocks !== 2 ||
inspectedBytes !== expectedTarBytes
) {
throw new Error('SSH relay TAR is truncated or lacks its two-block end marker')
}
const missing = tuple.entries.find((entry) => !seen.has(entry.path))
if (missing) {
throw new Error(`SSH relay TAR is missing a declared entry: ${missing.path}`)
}
}
return new Writable({
write(chunk: Buffer, _encoding, callback) {
try {
consume(chunk)
callback()
} catch (error) {
callback(error instanceof Error ? error : new Error(String(error)))
}
},
final(callback) {
try {
finish()
callback()
} catch (error) {
callback(error instanceof Error ? error : new Error(String(error)))
}
}
})
}
export async function inspectSshRelayTarBrotli({
archivePath,
tuple,
signal,
chunkBytes
}: {
archivePath: string
tuple: SelectedTuple
signal: AbortSignal
chunkBytes: number
}): Promise<void> {
// Why: a block-state inspector avoids retaining expanded TAR data before the separate write pass.
await pipeline(
createReadStream(archivePath, { highWaterMark: chunkBytes }),
createBrotliDecompress({ chunkSize: chunkBytes }),
inspectionSink(tuple),
{ signal }
)
}
@@ -1,6 +1,5 @@
import { createHash } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { lstat, readdir } from 'node:fs/promises'
import { lstat, open, readdir } from 'node:fs/promises'
import { join, relative, sep } from 'node:path'
import type { SshRelaySelectedArtifact } from './ssh-relay-artifact-selector'
@@ -18,12 +17,39 @@ async function hashFile(
): Promise<string> {
const digest = createHash('sha256')
let size = 0
for await (const chunk of createReadStream(path, { highWaterMark: chunkBytes, signal })) {
size += chunk.length
if (size > expectedSize) {
throw new Error('SSH relay extracted tree file exceeds its signed size')
const handle = await open(path, 'r')
try {
const before = await handle.stat({ bigint: true })
if (!before.isFile() || before.size !== BigInt(expectedSize)) {
throw new Error('SSH relay extracted tree file changed before hashing')
}
digest.update(chunk)
const buffer = Buffer.allocUnsafe(Math.min(chunkBytes, Math.max(expectedSize, 1)))
while (size < expectedSize) {
signal.throwIfAborted()
const { bytesRead } = await handle.read(
buffer,
0,
Math.min(buffer.length, expectedSize - size),
size
)
if (bytesRead === 0) {
break
}
size += bytesRead
digest.update(buffer.subarray(0, bytesRead))
}
const after = await handle.stat({ bigint: true })
if (
before.dev !== after.dev ||
before.ino !== after.ino ||
before.size !== after.size ||
before.mtimeNs !== after.mtimeNs ||
before.ctimeNs !== after.ctimeNs
) {
throw new Error('SSH relay extracted tree file changed while hashing')
}
} finally {
await handle.close()
}
if (size !== expectedSize) {
throw new Error('SSH relay extracted tree file size disagrees with its signed size')