mirror of
https://github.com/stablyai/orca.git
synced 2026-10-26 00:03:55 +00:00
* feat(remote): add a positional file read to the filesystem provider Following a growing remote file means re-reading it from the top on every poll: the relay exposes only whole-file reads, so tailing an append-only log over SSH costs O(size) per tick. Adds fs.readFileRange plus a rangedReadVersion capability, and an optional readFileRange on IFilesystemProvider -- matching how lstat/ supportsQuickOpenSearch already declare degradable capabilities. Three deliberate choices: - The relay loops until the requested length is satisfied or the file truly ends, and REJECTS an over-cap request rather than clamping it. A clamped read is indistinguishable from EOF, so a caller advancing a cursor by bytesRead would silently skip data. - Bytes cross the wire base64-encoded. A range boundary can split a UTF-8 sequence at either edge, and a utf-8 round trip would substitute U+FFFD and shift every subsequent offset. - The provider throws a typed FileRangeReadUnsupportedError against an older relay instead of quietly falling back to a whole-file read. A tailing caller issues several reads per snapshot, so a per-call fallback is quadratic; callers probe supportsFileRangeRead once and snapshot instead. The response is validated before use -- a byte count disagreeing with the payload would shift every downstream offset while looking like success. Terminal-artifact reads/writes move to their own module, mirroring the relay's existing fs-handler-terminal-artifact split; the provider was at the max-lines ceiling and this was the cohesive piece to extract. * fix(remote): size the ranged read to what the relay writer can deliver The 4 MiB cap was justified against MAX_MESSAGE_SIZE (16 MiB), but that is the frame DECODER bound. Responses are gated by the writer's admission budget: a frame over DISPATCHER_CONTROL_QUEUE_MAX_BYTES (1 MiB) is demoted to the legacy-response lane, which is refused once the producer queue passes 2 MiB. A 4 MiB window is ~5.46 MiB of base64, so it was never admissible -- it came back as an opaque ResponseOverCapacity (-33008), which is neither of the PR's typed errors, and above ~1.4 MiB the outcome depended on unrelated queued traffic. Cap at STREAM_CHUNK_SIZE (256 KiB), the house per-frame budget for file bytes, which stays in the control lane unconditionally. Also: - Hoist the cap and offset validation into src/shared/file-range-read.ts so the client rejects an out-of-contract request locally instead of paying a round trip for an error that does not survive the wire as a type. - Validate filePath in the relay handler; a missing one threw a TypeError out of expandTilde despite the comment claiming hand-validated params. - Collapse the two fs.getCapabilities probes onto one cached fetch per multiplexer. They read one document, so probing per feature spent an extra round trip per connection and duplicated the eviction logic. - Reuse readFullStreamChunk instead of a second copy of the short-read fill. - allocUnsafe the window; only subarray(0, bytesRead) escapes, so a tailing poll no longer memsets the whole window per call. - Plain methods for readFileRange/supportsFileRangeRead rather than constructor-assigned arrows; both are unconditional, unlike downloadFolder. Tests: cover the dispatch path and fs.getCapabilities (neither was exercised), param validation at both boundaries, EOF at and past the end, and a full-cap read over a real RelayDispatcher. The transport guard fails at 4 MiB with the real -33008. * test(remote): pin the ranged-read cap to real control-queue headroom The cap comment claimed a full-cap window stays in the control lane "unconditionally" and the guard test only asserted one frame fits the lane, so a raise to 384-768 KiB stayed green while two concurrent full-cap responses would already overflow the shared control queue -- which for a response closes the client. Pin the two-deep headroom and state the real bound, including that widening the cap is a wire change against a host still advertising rangedReadVersion 1. Also cover the two behaviours the suite claimed but did not exercise: a regular file answers a full-cap read in one syscall, so the fill loop was untested (both mutations of readFullStreamChunk stayed green), and the merged capability document made the abort-does-not-evict guard load-bearing without any test reaching it. * fix(remote): harden ranged-read validation and retry
239 lines
10 KiB
TypeScript
239 lines
10 KiB
TypeScript
import { tmpdir } from 'node:os'
|
|
import type { RelayDispatcher, RequestContext } from './dispatcher'
|
|
import type { RelayContext } from './context'
|
|
// Why: RelayContext is accepted in the constructor for protocol back-compat
|
|
// (see docs/relay-fs-allowlist-removal.md), but no longer consulted on FS ops.
|
|
import { expandTilde } from './context'
|
|
import { DEFAULT_MAX_RESULTS, searchWithRg } from './fs-handler-utils'
|
|
import { searchWithGitGrep } from './fs-handler-git-fallback'
|
|
import { ListFilesScanCoordinator } from './fs-list-files-scan-coordinator'
|
|
import { runListFilesScan } from './fs-list-files-fallback-chain'
|
|
import {
|
|
lstatRelayPath,
|
|
readRelayDir,
|
|
realpathRelayPath,
|
|
statRelayPath
|
|
} from './fs-path-metadata-requests'
|
|
import {
|
|
copyRelayPath,
|
|
createRelayDir,
|
|
createRelayDirNoClobber,
|
|
createRelayFile,
|
|
deleteRelayPath,
|
|
renameRelayPath,
|
|
renameRelayPathNoClobber,
|
|
writeRelayFile
|
|
} from './fs-path-mutation-requests'
|
|
import { buildExcludePathPrefixes } from '../shared/quick-open-filter'
|
|
import { readRelayFileContent, readRelayFileStreamMetadata } from './fs-handler-file-read'
|
|
import { readRelayFileRange } from './fs-handler-file-range'
|
|
import { FileRangeReadRequestError } from '../shared/file-range-read'
|
|
import {
|
|
readVerifiedTerminalArtifact,
|
|
writeVerifiedTerminalArtifact
|
|
} from './fs-handler-terminal-artifact'
|
|
import { RelayStreamRegistry } from './fs-stream-registry'
|
|
import { scanWorkspaceSpaceDirectory } from './workspace-space-scan'
|
|
import { RipgrepUnavailableError } from '../shared/ripgrep-process-availability'
|
|
import { RelayFilesystemWatchRegistry } from './relay-filesystem-watch-registry'
|
|
import type { RelayWatcherProcessPool } from './relay-watcher-process-pool'
|
|
|
|
export class FsHandler {
|
|
private dispatcher: RelayDispatcher
|
|
private watchRegistry: RelayFilesystemWatchRegistry
|
|
private streamRegistry = new RelayStreamRegistry()
|
|
private listFilesScans = new ListFilesScanCoordinator()
|
|
|
|
constructor(
|
|
dispatcher: RelayDispatcher,
|
|
_context: RelayContext,
|
|
watcherPool?: RelayWatcherProcessPool
|
|
) {
|
|
this.dispatcher = dispatcher
|
|
this.watchRegistry = new RelayFilesystemWatchRegistry(dispatcher, watcherPool)
|
|
this.registerHandlers()
|
|
this.dispatcher.onClientDetached?.(() => {
|
|
// Why: a detached client's fs.streamAck frames will never arrive; wake
|
|
// any pump parked on the ack window so it re-checks staleness and exits
|
|
// instead of stranding its open file handle.
|
|
this.streamRegistry.wakeAllAckWaiters()
|
|
})
|
|
}
|
|
|
|
getWatchRegistry(): RelayFilesystemWatchRegistry {
|
|
return this.watchRegistry
|
|
}
|
|
|
|
private registerHandlers(): void {
|
|
this.dispatcher.onRequest('fs.readDir', (p) => readRelayDir(p))
|
|
this.dispatcher.onRequest('fs.readFile', (p) => this.readFile(p))
|
|
this.dispatcher.onRequest('fs.readFileStream', (p, c) => this.readFileStream(p, c))
|
|
this.dispatcher.onRequest('fs.readFileRange', (p) => this.readFileRange(p))
|
|
this.dispatcher.onRequest('fs.readTerminalArtifact', (p) => this.readTerminalArtifact(p))
|
|
this.dispatcher.onRequest('fs.tempDir', () => this.tempDir())
|
|
this.dispatcher.onRequest('fs.writeFile', (p) => writeRelayFile(p))
|
|
this.dispatcher.onRequest('fs.writeTerminalArtifact', (p) => this.writeTerminalArtifact(p))
|
|
this.dispatcher.onRequest('fs.stat', (p) => statRelayPath(p))
|
|
this.dispatcher.onRequest('fs.lstat', (p) => lstatRelayPath(p))
|
|
this.dispatcher.onRequest('fs.deletePath', (p) => deleteRelayPath(p, this.watchRegistry))
|
|
this.dispatcher.onRequest('fs.createFile', (p) => createRelayFile(p))
|
|
this.dispatcher.onRequest('fs.createDir', (p) => createRelayDir(p))
|
|
this.dispatcher.onRequest('fs.createDirNoClobber', (p) => createRelayDirNoClobber(p))
|
|
this.dispatcher.onRequest('fs.rename', (p) => renameRelayPath(p))
|
|
this.dispatcher.onRequest('fs.renameNoClobber', (p) => renameRelayPathNoClobber(p))
|
|
this.dispatcher.onRequest('fs.copy', (p) => copyRelayPath(p))
|
|
this.dispatcher.onRequest('fs.realpath', (p) => realpathRelayPath(p))
|
|
this.dispatcher.onRequest('fs.search', (p) => this.search(p))
|
|
this.dispatcher.onRequest('fs.getCapabilities', async () => ({
|
|
quickOpenSearchVersion: 1,
|
|
rangedReadVersion: 1
|
|
}))
|
|
this.dispatcher.onRequest('fs.listFiles', (p, c) => this.listFiles(p, c))
|
|
this.dispatcher.onRequest('fs.workspaceSpaceScan', (p, c) => this.workspaceSpaceScan(p, c))
|
|
this.dispatcher.onRequest('fs.watch', (p, context) =>
|
|
this.watchRegistry.watch(
|
|
expandTilde(p.rootPath as string),
|
|
context,
|
|
typeof p.watchId === 'number' && Number.isSafeInteger(p.watchId) ? p.watchId : undefined
|
|
)
|
|
)
|
|
this.dispatcher.onRequest('fs.unwatchAndWait', (p, context) =>
|
|
this.watchRegistry.unwatchAndWait(expandTilde(p.rootPath as string), context)
|
|
)
|
|
this.dispatcher.onNotification('fs.unwatch', (p, context) =>
|
|
this.watchRegistry.unwatch(expandTilde(p.rootPath as string), context)
|
|
)
|
|
this.dispatcher.onNotification('fs.cancelStream', (p) => this.cancelStream(p))
|
|
this.dispatcher.onNotification('fs.streamAck', (p) => this.streamAck(p))
|
|
}
|
|
|
|
private async readFile(params: Record<string, unknown>) {
|
|
const filePath = expandTilde(params.filePath as string)
|
|
return readRelayFileContent(filePath)
|
|
}
|
|
|
|
// Why hand-checked: relay params arrive as raw casts with no schema. The
|
|
// offsets are validated inside readRelayFileRange, next to the read syscall.
|
|
private async readFileRange(params: Record<string, unknown>) {
|
|
if (typeof params.filePath !== 'string' || params.filePath.length === 0) {
|
|
throw new FileRangeReadRequestError('fs.readFileRange requires a filePath')
|
|
}
|
|
return readRelayFileRange(expandTilde(params.filePath), params.position, params.length)
|
|
}
|
|
|
|
private async readTerminalArtifact(params: Record<string, unknown>) {
|
|
return readVerifiedTerminalArtifact({
|
|
...params,
|
|
filePath: expandTilde(params.filePath as string)
|
|
})
|
|
}
|
|
|
|
private async readFileStream(params: Record<string, unknown>, context?: RequestContext) {
|
|
const filePath = expandTilde(params.filePath as string)
|
|
const ctx = context ?? { clientId: 0, isStale: () => false }
|
|
return readRelayFileStreamMetadata(filePath, this.dispatcher, this.streamRegistry, ctx, {
|
|
// Why: only target the requesting client when the dispatcher actually
|
|
// routed this request (context present) — direct-call tests and legacy
|
|
// paths keep broadcast semantics.
|
|
...(context ? { clientId: context.clientId } : {}),
|
|
paceWithAcks: params.flowControl === 'ack'
|
|
})
|
|
}
|
|
|
|
private async tempDir(): Promise<string> {
|
|
return tmpdir()
|
|
}
|
|
|
|
private cancelStream(params: Record<string, unknown>): void {
|
|
const streamId = params.streamId as number | undefined
|
|
if (typeof streamId === 'number') {
|
|
this.streamRegistry.abort(streamId)
|
|
}
|
|
}
|
|
|
|
private streamAck(params: Record<string, unknown>): void {
|
|
const streamId = params.streamId as number | undefined
|
|
const seq = params.seq as number | undefined
|
|
if (typeof streamId === 'number' && typeof seq === 'number') {
|
|
this.streamRegistry.recordAck(streamId, seq)
|
|
}
|
|
}
|
|
|
|
private async writeTerminalArtifact(params: Record<string, unknown>) {
|
|
return writeVerifiedTerminalArtifact({
|
|
...params,
|
|
filePath: expandTilde(params.filePath as string)
|
|
})
|
|
}
|
|
|
|
private async search(params: Record<string, unknown>) {
|
|
const query = params.query as string
|
|
const rootPath = expandTilde(params.rootPath as string)
|
|
const caseSensitive = params.caseSensitive as boolean | undefined
|
|
const wholeWord = params.wholeWord as boolean | undefined
|
|
const useRegex = params.useRegex as boolean | undefined
|
|
const includePattern = params.includePattern as string | undefined
|
|
const excludePattern = params.excludePattern as string | undefined
|
|
const maxResults = Math.min(
|
|
(params.maxResults as number) || DEFAULT_MAX_RESULTS,
|
|
DEFAULT_MAX_RESULTS
|
|
)
|
|
|
|
const options = {
|
|
caseSensitive,
|
|
wholeWord,
|
|
useRegex,
|
|
includePattern,
|
|
excludePattern,
|
|
maxResults
|
|
}
|
|
try {
|
|
return await searchWithRg(rootPath, query, options)
|
|
} catch (error) {
|
|
if (!(error instanceof RipgrepUnavailableError)) {
|
|
throw error
|
|
}
|
|
return searchWithGitGrep(rootPath, query, options)
|
|
}
|
|
}
|
|
|
|
private listFiles(params: Record<string, unknown>, context?: RequestContext): Promise<string[]> {
|
|
const rootPath = expandTilde(params.rootPath as string)
|
|
const maxResults =
|
|
typeof params.maxResults === 'number' &&
|
|
Number.isInteger(params.maxResults) &&
|
|
params.maxResults > 0
|
|
? Math.min(params.maxResults, 20_001)
|
|
: undefined
|
|
const searchQuery =
|
|
typeof params.searchQuery === 'string' && params.searchQuery.trim().length > 0
|
|
? params.searchQuery
|
|
: undefined
|
|
// Why: the main-to-relay RPC adds excludePaths so nested linked worktrees
|
|
// don't get double-scanned. The shared helper validates the shape and
|
|
// normalizes into root-relative prefixes; malformed input yields [] so
|
|
// the request still succeeds (older apps omit the field entirely).
|
|
const excludePathPrefixes = buildExcludePathPrefixes(rootPath, params.excludePaths)
|
|
// Why #7721: full-tree scans are the relay's most expensive request; the
|
|
// coordinator caps them at one per client, coalescing duplicates and
|
|
// aborting a stale scan when the workspace changes or the host cancels.
|
|
return this.listFilesScans.run({
|
|
clientId: context?.clientId ?? 0,
|
|
key: JSON.stringify([rootPath, excludePathPrefixes, maxResults, searchQuery]),
|
|
signal: context?.signal,
|
|
start: (signal) =>
|
|
runListFilesScan(rootPath, excludePathPrefixes, signal, maxResults, searchQuery)
|
|
})
|
|
}
|
|
|
|
private async workspaceSpaceScan(params: Record<string, unknown>, context: RequestContext) {
|
|
const rootPath = expandTilde(params.rootPath as string)
|
|
return scanWorkspaceSpaceDirectory(rootPath, context)
|
|
}
|
|
|
|
dispose(): void {
|
|
this.watchRegistry.dispose()
|
|
void this.streamRegistry.disposeAll()
|
|
}
|
|
}
|