Files
orca/src/relay/fs-handler.ts
T
Brennan Benson 6415511a82 feat(remote): add a positional file read to the filesystem provider (#15517)
* feat(remote): add a positional file read to the filesystem provider

Following a growing remote file means re-reading it from the top on every
poll: the relay exposes only whole-file reads, so tailing an append-only log
over SSH costs O(size) per tick.

Adds fs.readFileRange plus a rangedReadVersion capability, and an optional
readFileRange on IFilesystemProvider -- matching how lstat/
supportsQuickOpenSearch already declare degradable capabilities.

Three deliberate choices:

- The relay loops until the requested length is satisfied or the file truly
  ends, and REJECTS an over-cap request rather than clamping it. A clamped
  read is indistinguishable from EOF, so a caller advancing a cursor by
  bytesRead would silently skip data.
- Bytes cross the wire base64-encoded. A range boundary can split a UTF-8
  sequence at either edge, and a utf-8 round trip would substitute U+FFFD and
  shift every subsequent offset.
- The provider throws a typed FileRangeReadUnsupportedError against an older
  relay instead of quietly falling back to a whole-file read. A tailing caller
  issues several reads per snapshot, so a per-call fallback is quadratic;
  callers probe supportsFileRangeRead once and snapshot instead.

The response is validated before use -- a byte count disagreeing with the
payload would shift every downstream offset while looking like success.

Terminal-artifact reads/writes move to their own module, mirroring the relay's
existing fs-handler-terminal-artifact split; the provider was at the max-lines
ceiling and this was the cohesive piece to extract.

* fix(remote): size the ranged read to what the relay writer can deliver

The 4 MiB cap was justified against MAX_MESSAGE_SIZE (16 MiB), but that is
the frame DECODER bound. Responses are gated by the writer's admission
budget: a frame over DISPATCHER_CONTROL_QUEUE_MAX_BYTES (1 MiB) is demoted
to the legacy-response lane, which is refused once the producer queue passes
2 MiB. A 4 MiB window is ~5.46 MiB of base64, so it was never admissible --
it came back as an opaque ResponseOverCapacity (-33008), which is neither of
the PR's typed errors, and above ~1.4 MiB the outcome depended on unrelated
queued traffic. Cap at STREAM_CHUNK_SIZE (256 KiB), the house per-frame
budget for file bytes, which stays in the control lane unconditionally.

Also:
- Hoist the cap and offset validation into src/shared/file-range-read.ts so
  the client rejects an out-of-contract request locally instead of paying a
  round trip for an error that does not survive the wire as a type.
- Validate filePath in the relay handler; a missing one threw a TypeError
  out of expandTilde despite the comment claiming hand-validated params.
- Collapse the two fs.getCapabilities probes onto one cached fetch per
  multiplexer. They read one document, so probing per feature spent an extra
  round trip per connection and duplicated the eviction logic.
- Reuse readFullStreamChunk instead of a second copy of the short-read fill.
- allocUnsafe the window; only subarray(0, bytesRead) escapes, so a tailing
  poll no longer memsets the whole window per call.
- Plain methods for readFileRange/supportsFileRangeRead rather than
  constructor-assigned arrows; both are unconditional, unlike downloadFolder.

Tests: cover the dispatch path and fs.getCapabilities (neither was
exercised), param validation at both boundaries, EOF at and past the end,
and a full-cap read over a real RelayDispatcher. The transport guard fails
at 4 MiB with the real -33008.

* test(remote): pin the ranged-read cap to real control-queue headroom

The cap comment claimed a full-cap window stays in the control lane
"unconditionally" and the guard test only asserted one frame fits the
lane, so a raise to 384-768 KiB stayed green while two concurrent
full-cap responses would already overflow the shared control queue --
which for a response closes the client. Pin the two-deep headroom and
state the real bound, including that widening the cap is a wire change
against a host still advertising rangedReadVersion 1.

Also cover the two behaviours the suite claimed but did not exercise: a
regular file answers a full-cap read in one syscall, so the fill loop
was untested (both mutations of readFullStreamChunk stayed green), and
the merged capability document made the abort-does-not-evict guard
load-bearing without any test reaching it.

* fix(remote): harden ranged-read validation and retry
2026-08-19 16:45:42 -07:00

239 lines
10 KiB
TypeScript

import { tmpdir } from 'node:os'
import type { RelayDispatcher, RequestContext } from './dispatcher'
import type { RelayContext } from './context'
// Why: RelayContext is accepted in the constructor for protocol back-compat
// (see docs/relay-fs-allowlist-removal.md), but no longer consulted on FS ops.
import { expandTilde } from './context'
import { DEFAULT_MAX_RESULTS, searchWithRg } from './fs-handler-utils'
import { searchWithGitGrep } from './fs-handler-git-fallback'
import { ListFilesScanCoordinator } from './fs-list-files-scan-coordinator'
import { runListFilesScan } from './fs-list-files-fallback-chain'
import {
lstatRelayPath,
readRelayDir,
realpathRelayPath,
statRelayPath
} from './fs-path-metadata-requests'
import {
copyRelayPath,
createRelayDir,
createRelayDirNoClobber,
createRelayFile,
deleteRelayPath,
renameRelayPath,
renameRelayPathNoClobber,
writeRelayFile
} from './fs-path-mutation-requests'
import { buildExcludePathPrefixes } from '../shared/quick-open-filter'
import { readRelayFileContent, readRelayFileStreamMetadata } from './fs-handler-file-read'
import { readRelayFileRange } from './fs-handler-file-range'
import { FileRangeReadRequestError } from '../shared/file-range-read'
import {
readVerifiedTerminalArtifact,
writeVerifiedTerminalArtifact
} from './fs-handler-terminal-artifact'
import { RelayStreamRegistry } from './fs-stream-registry'
import { scanWorkspaceSpaceDirectory } from './workspace-space-scan'
import { RipgrepUnavailableError } from '../shared/ripgrep-process-availability'
import { RelayFilesystemWatchRegistry } from './relay-filesystem-watch-registry'
import type { RelayWatcherProcessPool } from './relay-watcher-process-pool'
export class FsHandler {
private dispatcher: RelayDispatcher
private watchRegistry: RelayFilesystemWatchRegistry
private streamRegistry = new RelayStreamRegistry()
private listFilesScans = new ListFilesScanCoordinator()
constructor(
dispatcher: RelayDispatcher,
_context: RelayContext,
watcherPool?: RelayWatcherProcessPool
) {
this.dispatcher = dispatcher
this.watchRegistry = new RelayFilesystemWatchRegistry(dispatcher, watcherPool)
this.registerHandlers()
this.dispatcher.onClientDetached?.(() => {
// Why: a detached client's fs.streamAck frames will never arrive; wake
// any pump parked on the ack window so it re-checks staleness and exits
// instead of stranding its open file handle.
this.streamRegistry.wakeAllAckWaiters()
})
}
getWatchRegistry(): RelayFilesystemWatchRegistry {
return this.watchRegistry
}
private registerHandlers(): void {
this.dispatcher.onRequest('fs.readDir', (p) => readRelayDir(p))
this.dispatcher.onRequest('fs.readFile', (p) => this.readFile(p))
this.dispatcher.onRequest('fs.readFileStream', (p, c) => this.readFileStream(p, c))
this.dispatcher.onRequest('fs.readFileRange', (p) => this.readFileRange(p))
this.dispatcher.onRequest('fs.readTerminalArtifact', (p) => this.readTerminalArtifact(p))
this.dispatcher.onRequest('fs.tempDir', () => this.tempDir())
this.dispatcher.onRequest('fs.writeFile', (p) => writeRelayFile(p))
this.dispatcher.onRequest('fs.writeTerminalArtifact', (p) => this.writeTerminalArtifact(p))
this.dispatcher.onRequest('fs.stat', (p) => statRelayPath(p))
this.dispatcher.onRequest('fs.lstat', (p) => lstatRelayPath(p))
this.dispatcher.onRequest('fs.deletePath', (p) => deleteRelayPath(p, this.watchRegistry))
this.dispatcher.onRequest('fs.createFile', (p) => createRelayFile(p))
this.dispatcher.onRequest('fs.createDir', (p) => createRelayDir(p))
this.dispatcher.onRequest('fs.createDirNoClobber', (p) => createRelayDirNoClobber(p))
this.dispatcher.onRequest('fs.rename', (p) => renameRelayPath(p))
this.dispatcher.onRequest('fs.renameNoClobber', (p) => renameRelayPathNoClobber(p))
this.dispatcher.onRequest('fs.copy', (p) => copyRelayPath(p))
this.dispatcher.onRequest('fs.realpath', (p) => realpathRelayPath(p))
this.dispatcher.onRequest('fs.search', (p) => this.search(p))
this.dispatcher.onRequest('fs.getCapabilities', async () => ({
quickOpenSearchVersion: 1,
rangedReadVersion: 1
}))
this.dispatcher.onRequest('fs.listFiles', (p, c) => this.listFiles(p, c))
this.dispatcher.onRequest('fs.workspaceSpaceScan', (p, c) => this.workspaceSpaceScan(p, c))
this.dispatcher.onRequest('fs.watch', (p, context) =>
this.watchRegistry.watch(
expandTilde(p.rootPath as string),
context,
typeof p.watchId === 'number' && Number.isSafeInteger(p.watchId) ? p.watchId : undefined
)
)
this.dispatcher.onRequest('fs.unwatchAndWait', (p, context) =>
this.watchRegistry.unwatchAndWait(expandTilde(p.rootPath as string), context)
)
this.dispatcher.onNotification('fs.unwatch', (p, context) =>
this.watchRegistry.unwatch(expandTilde(p.rootPath as string), context)
)
this.dispatcher.onNotification('fs.cancelStream', (p) => this.cancelStream(p))
this.dispatcher.onNotification('fs.streamAck', (p) => this.streamAck(p))
}
private async readFile(params: Record<string, unknown>) {
const filePath = expandTilde(params.filePath as string)
return readRelayFileContent(filePath)
}
// Why hand-checked: relay params arrive as raw casts with no schema. The
// offsets are validated inside readRelayFileRange, next to the read syscall.
private async readFileRange(params: Record<string, unknown>) {
if (typeof params.filePath !== 'string' || params.filePath.length === 0) {
throw new FileRangeReadRequestError('fs.readFileRange requires a filePath')
}
return readRelayFileRange(expandTilde(params.filePath), params.position, params.length)
}
private async readTerminalArtifact(params: Record<string, unknown>) {
return readVerifiedTerminalArtifact({
...params,
filePath: expandTilde(params.filePath as string)
})
}
private async readFileStream(params: Record<string, unknown>, context?: RequestContext) {
const filePath = expandTilde(params.filePath as string)
const ctx = context ?? { clientId: 0, isStale: () => false }
return readRelayFileStreamMetadata(filePath, this.dispatcher, this.streamRegistry, ctx, {
// Why: only target the requesting client when the dispatcher actually
// routed this request (context present) — direct-call tests and legacy
// paths keep broadcast semantics.
...(context ? { clientId: context.clientId } : {}),
paceWithAcks: params.flowControl === 'ack'
})
}
private async tempDir(): Promise<string> {
return tmpdir()
}
private cancelStream(params: Record<string, unknown>): void {
const streamId = params.streamId as number | undefined
if (typeof streamId === 'number') {
this.streamRegistry.abort(streamId)
}
}
private streamAck(params: Record<string, unknown>): void {
const streamId = params.streamId as number | undefined
const seq = params.seq as number | undefined
if (typeof streamId === 'number' && typeof seq === 'number') {
this.streamRegistry.recordAck(streamId, seq)
}
}
private async writeTerminalArtifact(params: Record<string, unknown>) {
return writeVerifiedTerminalArtifact({
...params,
filePath: expandTilde(params.filePath as string)
})
}
private async search(params: Record<string, unknown>) {
const query = params.query as string
const rootPath = expandTilde(params.rootPath as string)
const caseSensitive = params.caseSensitive as boolean | undefined
const wholeWord = params.wholeWord as boolean | undefined
const useRegex = params.useRegex as boolean | undefined
const includePattern = params.includePattern as string | undefined
const excludePattern = params.excludePattern as string | undefined
const maxResults = Math.min(
(params.maxResults as number) || DEFAULT_MAX_RESULTS,
DEFAULT_MAX_RESULTS
)
const options = {
caseSensitive,
wholeWord,
useRegex,
includePattern,
excludePattern,
maxResults
}
try {
return await searchWithRg(rootPath, query, options)
} catch (error) {
if (!(error instanceof RipgrepUnavailableError)) {
throw error
}
return searchWithGitGrep(rootPath, query, options)
}
}
private listFiles(params: Record<string, unknown>, context?: RequestContext): Promise<string[]> {
const rootPath = expandTilde(params.rootPath as string)
const maxResults =
typeof params.maxResults === 'number' &&
Number.isInteger(params.maxResults) &&
params.maxResults > 0
? Math.min(params.maxResults, 20_001)
: undefined
const searchQuery =
typeof params.searchQuery === 'string' && params.searchQuery.trim().length > 0
? params.searchQuery
: undefined
// Why: the main-to-relay RPC adds excludePaths so nested linked worktrees
// don't get double-scanned. The shared helper validates the shape and
// normalizes into root-relative prefixes; malformed input yields [] so
// the request still succeeds (older apps omit the field entirely).
const excludePathPrefixes = buildExcludePathPrefixes(rootPath, params.excludePaths)
// Why #7721: full-tree scans are the relay's most expensive request; the
// coordinator caps them at one per client, coalescing duplicates and
// aborting a stale scan when the workspace changes or the host cancels.
return this.listFilesScans.run({
clientId: context?.clientId ?? 0,
key: JSON.stringify([rootPath, excludePathPrefixes, maxResults, searchQuery]),
signal: context?.signal,
start: (signal) =>
runListFilesScan(rootPath, excludePathPrefixes, signal, maxResults, searchQuery)
})
}
private async workspaceSpaceScan(params: Record<string, unknown>, context: RequestContext) {
const rootPath = expandTilde(params.rootPath as string)
return scanWorkspaceSpaceDirectory(rootPath, context)
}
dispose(): void {
this.watchRegistry.dispose()
void this.streamRegistry.disposeAll()
}
}