From 1617ff32eff18756c2a39d63f9512cbe7ac0f692 Mon Sep 17 00:00:00 2001 From: Brennan Benson <79079362+brennanb2025@users.noreply.github.com> Date: Wed, 7 Oct 2026 14:21:05 -0700 Subject: [PATCH] feat(mobile): show chat visuals inline in the phone's native chat (#26071) * feat(native-chat): visual directive grammar and host read for a chat's visuals folder A shared grammar for the ::orca-visual{file="..." title="..."} reply line, the per-chat visuals folder location on the owning host, and the agentSession.readVisual runtime method that reads one visual with lexical and canonical containment, a 512 KiB bounded read and UTF-8 refusal. * feat(native-chat): shared frame document for chat visuals One string builder every client wraps a visual's HTML with: the policy (CDN assets only, no fetch, frames, workers, forms or base rewrites), the theme variables, and a prelude that reports height, routes links to the parent and refuses navigation. Also the validated frame-to-parent message reader and the live theme message. * feat(mobile): render native-chat visuals inline in the phone's chat A finished assistant reply's ::orca-visual line now shows the visual inline, read from the chat's owning host through agentSession.readVisual. The visual runs in an opaque sandboxed child of a trusted host document inside the WebView; the app accepts only a token-checked height and an http(s) link opened under user activation. Navigation away from the visual's own document is refused, a dead web process reloads once, and the visual opens full screen. The hybrid shell's page renders it sealed. * feat(native-chat): render chat visuals inline and in the right sidebar Native-chat assistant replies render a ::orca-visual{...} line as the chat's HTML visual in an opaque, scripts-only sandboxed frame: CSP first, the host frame navigation guard registered before content runs, live theme without a reload, fitted height, links opened in the viewer's browser only from a real gesture, lazy mount, and one muted line when the visual cannot be shown. Open in sidebar shows the same frame in the right sidebar, widened while it is open and restored after. * fix(mobile): chat visuals use the shared shell; only the host page may message the app Review round 1: - use PR 1's shared visual shell and height governor instead of a second builder; the app decides heights and pushes them to the host page - react-native-webview patch: the message channel accepts only string messages from the main frame on iOS and Android, so a visual in its sandboxed child cannot reach it (or crash Android with a non-string) - structured replies grow in place: while a turn works, a row holds back a directive still being typed at its tail; finished lines mount - links need child focus + activation, one per activation window - a refused read takes the visual down and drops its cached bytes - in-page anchors load; text spelling a placeholder renders no visuals * refactor(native-chat): move the visual height governor to src/shared The phone bundles only src/shared, so the governor the desktop frame uses moves there unchanged and the mobile frame shares it. * fix(native-chat): point the desktop and mobile frames at the moved governor * fix(mobile): review round 2 for chat visuals - host page relays only messages on the visual's own channel, as its own copy, so a visual cannot push oversized fields through it; relay rate halved; a link is validated before it uses up the link window - the frame denies camera, microphone, geolocation, clipboard and display capture; iOS media capture requests are denied - react-native-webview patch: the iOS history-shim handler also accepts only main-frame string messages - only the newest assistant row of a working turn holds back an unfinished directive; earlier finished rows show their visuals - an error reply or older host is not a verdict: it keeps a visual on screen and its cache, and retries; only a host refusal takes it down * fix(mobile): a drag that starts on an inline visual scrolls the chat iOS gives a scrollable frame its own scroll view, which took the drag; the inline frame is sized to its content, so it no longer scrolls. Full screen still does. * fix(mobile): review round 3 for chat visuals - hold back only the last block of the newest assistant row, and not while a question or approval is open, so a visual followed by a tool call or a pending question shows at once - an older host (method_not_found) reads as unavailable without retries - drop a stray @pnpm/exe lockfile block; only the patch hash changes * test(mobile): list the visual frame's web sibling; mock it in the prompt-controller harness * fix(mobile): an older desktop's mobile allowlist refusal reads as unavailable at once * fix(native-chat): visual CI fixes, shared height governor, live-turn streaming hold Registers agentSession.readVisual from the methods index so the structured method file stays under its line budget, replaces reflective reads with checked narrowing, moves the pure height governor to src/shared for mobile, and holds a half-written directive tail while the turn works (structured text rows carry no running state). * fix(native-chat): harden the visual read and link opening Re-checks after the open that the chat's visuals folder is still the real directory at Orca's path, reports unexpected filesystem faults by code without host paths, and lets one click in a visual open at most one page. * fix(native-chat): keep visual lines out of plain-text reply surfaces; review fixes One shared helper drops visual lines (outside fenced code) from reply text where it becomes plain text: the structured status summary that feeds the sidebar row, dashboard, notifications, phone rows and handoffs, and AI Vault reply previews. Review fixes: height also counts a pinned body's overflow, only the live frontier row holds a half-written visual line, the runaway-height stop needs the same step repeated, and any host refusal evicts the cached revision. * fix(native-chat): resolve the visuals folder without the removed journal-paths helper Main removed the per-chat journal paths and the journal database's state directory; the visuals folder keeps the same sha256 layout on its own and the read method uses the profile state directory the chat host is opened in. * fix(mobile): keep visual lines out of the worktree row and copied message text A reply's ::orca-visual line renders only in the transcript. The worktree list's agent row and the message actions sheet's copy text now drop it with the shared helper; a reply that is only a visual falls back to the prompt, as an empty one does. * fix(native-chat): review round 2 fixes; copy a reply without its visual lines Reply previews in Agent Session History drop visual lines per text part before lines are folded; the frame adds a body's overflow only when the body really overflows; fence tracking follows CommonMark closers and openers; the copy button copies a reply without visual lines; a coded read fault keeps its cause. * fix(native-chat): update the frame's theme ref after render; read the visuals folder pair at once * test(native-chat): declare agentSession.readVisual on the cross-version agent-session surface * fix(native-chat): copying a reply keeps its code blocks and indentation Removing visual lines now closes only the gap each removal leaves, instead of collapsing blank lines across the whole reply and trimming its indentation; the visuals folder is checked parent first again so a broken path answers the same way every time. --- .../react-native-webview@13.16.2.patch | 43 ++++ mobile/pnpm-lock.yaml | 8 +- mobile/src/components/MobileMarkdown.tsx | 21 +- mobile/src/components/WorktreeAgentRow.tsx | 11 +- .../WorktreeAgentRow.visual-line.test.tsx | 59 ++++++ mobile/src/components/inline-script-json.ts | 17 ++ .../src/components/mobile-markdown-parser.ts | 19 +- .../mobile-markdown-visual-lines.test.ts | 90 ++++++++ .../mobile-markdown-visual-lines.ts | 69 ++++++ .../components/pr-sidebar/MermaidDiagram.tsx | 14 +- .../components/use-mobile-markdown-blocks.ts | 28 +++ .../src/session/MobileNativeChatMessage.tsx | 33 ++- .../session/MobileNativeChatOverlay.test.ts | 1 + .../src/session/MobileNativeChatOverlay.tsx | 125 +++++------ mobile/src/session/MobileNativeChatVisual.tsx | 189 +++++++++++++++++ .../session/MobileNativeChatVisualFrame.tsx | 200 ++++++++++++++++++ .../MobileNativeChatVisualFrame.web.tsx | 59 ++++++ .../__mocks__/mobile-prompt-controller.ts | 1 + .../mobile-native-chat-controller-contract.ts | 3 + ...chat-merged-snapshot-parity-hooks.test.tsx | 1 + ...ile-native-chat-message-plain-text.test.ts | 8 + .../mobile-native-chat-message-plain-text.ts | 4 +- ...mobile-native-chat-message-visuals.test.ts | 154 ++++++++++++++ .../mobile-native-chat-visual-bridge.test.ts | 104 +++++++++ .../mobile-native-chat-visual-bridge.ts | 51 +++++ .../mobile-native-chat-visual-context.ts | 15 ++ ...mobile-native-chat-visual-host-document.ts | 132 ++++++++++++ .../mobile-native-chat-visual-read.test.ts | 200 ++++++++++++++++++ .../session/mobile-native-chat-visual-read.ts | 169 +++++++++++++++ .../mobile-native-chat-visual-theme.ts | 40 ++++ .../use-mobile-native-chat-controller.ts | 1 + ...-chat-turn-disclosure-growing-row.test.tsx | 49 +++++ .../use-mobile-native-chat-turn-disclosure.ts | 9 + .../use-mobile-native-chat-visual.test.tsx | 92 ++++++++ .../session/use-mobile-native-chat-visual.ts | 87 ++++++++ .../use-mobile-structured-agent-session.ts | 9 + mobile/web-entry/web-overrides.json | 4 + src/main/runtime/mobile-rpc-allowlist.test.ts | 1 + ...untime-rpc-mobile-agent-session-methods.ts | 2 + 39 files changed, 2033 insertions(+), 89 deletions(-) create mode 100644 mobile/src/components/WorktreeAgentRow.visual-line.test.tsx create mode 100644 mobile/src/components/inline-script-json.ts create mode 100644 mobile/src/components/mobile-markdown-visual-lines.test.ts create mode 100644 mobile/src/components/mobile-markdown-visual-lines.ts create mode 100644 mobile/src/components/use-mobile-markdown-blocks.ts create mode 100644 mobile/src/session/MobileNativeChatVisual.tsx create mode 100644 mobile/src/session/MobileNativeChatVisualFrame.tsx create mode 100644 mobile/src/session/MobileNativeChatVisualFrame.web.tsx create mode 100644 mobile/src/session/mobile-native-chat-message-visuals.test.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-bridge.test.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-bridge.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-context.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-host-document.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-read.test.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-read.ts create mode 100644 mobile/src/session/mobile-native-chat-visual-theme.ts create mode 100644 mobile/src/session/use-mobile-native-chat-turn-disclosure-growing-row.test.tsx create mode 100644 mobile/src/session/use-mobile-native-chat-visual.test.tsx create mode 100644 mobile/src/session/use-mobile-native-chat-visual.ts diff --git a/mobile/patches/react-native-webview@13.16.2.patch b/mobile/patches/react-native-webview@13.16.2.patch index 963f8884f38..574cddd8b8d 100644 --- a/mobile/patches/react-native-webview@13.16.2.patch +++ b/mobile/patches/react-native-webview@13.16.2.patch @@ -1,3 +1,46 @@ +diff --git a/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java b/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java +index 80c68034b193a47bedb56ca527352de9548e8ac6..977c068a74c88c0322fca2a4bbffab8bc1ba719d 100644 +--- a/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java ++++ b/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java +@@ -253,6 +253,12 @@ public class RNCWebView extends WebView implements LifecycleEventListener { + this.bridgeListener = new WebViewCompat.WebMessageListener() { + @Override + public void onPostMessage(@NonNull WebView view, @NonNull WebMessageCompat message, @NonNull Uri sourceOrigin, boolean isMainFrame, @NonNull JavaScriptReplyProxy replyProxy) { ++ // Orca: only the page's own document may message the app. A subframe (for example agent ++ // HTML in a sandboxed iframe) reaches this listener too, and getData() throws for a ++ // non-string message. ++ if (!isMainFrame || message.getType() != WebMessageCompat.TYPE_STRING) { ++ return; ++ } + RNCWebView.this.onMessage(message.getData(), sourceOrigin.toString()); + } + }; +diff --git a/apple/RNCWebViewImpl.m b/apple/RNCWebViewImpl.m +index 7f5c24d6e2cfb26eddbc513bf17a81c24a7273e2..d70cb58a3c81cb22bd549a2c37340ba53853cfcf 100644 +--- a/apple/RNCWebViewImpl.m ++++ b/apple/RNCWebViewImpl.m +@@ -776,12 +776,21 @@ - (void)userContentController:(WKUserContentController *)userContentController + didReceiveScriptMessage:(WKScriptMessage *)message + { + if ([message.name isEqualToString:HistoryShimName]) { ++ // Orca: the history shim is the page's own script; a subframe or non-string body is not it. ++ if (!message.frameInfo.isMainFrame || ![message.body isKindOfClass:[NSString class]]) { ++ return; ++ } + if (_onLoadingFinish) { + NSMutableDictionary *event = [self baseEvent]; + [event addEntriesFromDictionary: @{@"navigationType": message.body}]; + _onLoadingFinish(event); + } + } else if ([message.name isEqualToString:MessageHandlerName]) { ++ // Orca: only the page's own document may message the app. A subframe (for example agent HTML in ++ // a sandboxed iframe) reaches this handler too, and a non-string body is not a message. ++ if (!message.frameInfo.isMainFrame || ![message.body isKindOfClass:[NSString class]]) { ++ return; ++ } + if (_onMessage) { + NSMutableDictionary *event = [self baseEvent]; + [event addEntriesFromDictionary: @{@"data": message.body}]; diff --git a/index.d.ts b/index.d.ts index 10b9c145b920a6a2ba1fb09bd72150328fafe211..626a67a3d6aca587ee9e74a52b799a2c50b4c52b 100644 --- a/index.d.ts diff --git a/mobile/pnpm-lock.yaml b/mobile/pnpm-lock.yaml index ee08c1422d2..6574a53c1cd 100644 --- a/mobile/pnpm-lock.yaml +++ b/mobile/pnpm-lock.yaml @@ -172,7 +172,7 @@ overrides: patchedDependencies: expo-notifications@55.0.27: ce20843a3daad4185d7e8571788fa323ba4d11984936188790858650a61749c0 query-string@9.5.1: 8624f1beedd0705bfdaf287519ed6363662a0c96f01fd594508b34583bf30d8b - react-native-webview@13.16.2: de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27 + react-native-webview@13.16.2: 55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06 react-native@0.83.10: 44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d importers: @@ -310,7 +310,7 @@ importers: version: 0.21.3(react-dom@19.2.8(react@19.2.8))(react@19.2.8) react-native-webview: specifier: 13.16.2 - version: 13.16.2(patch_hash=de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) + version: 13.16.2(patch_hash=55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) react-native-worklets: specifier: ^0.8.3 version: 0.8.3(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) @@ -12228,7 +12228,7 @@ snapshots: optionalDependencies: '@expo/dom-webview': 55.0.6(expo@55.0.31)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) '@expo/metro-runtime': 55.0.12(@expo/dom-webview@55.0.6)(expo@55.0.31)(react-dom@19.2.8(react@19.2.8))(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) - react-native-webview: 13.16.2(patch_hash=de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) + react-native-webview: 13.16.2(patch_hash=55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8) transitivePeerDependencies: - '@babel/core' - bufferutil @@ -14343,7 +14343,7 @@ snapshots: transitivePeerDependencies: - encoding - react-native-webview@13.16.2(patch_hash=de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8): + react-native-webview@13.16.2(patch_hash=55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8): dependencies: '@typescript/native-preview': 7.0.0-dev.20260707.2 escape-string-regexp: 4.0.0 diff --git a/mobile/src/components/MobileMarkdown.tsx b/mobile/src/components/MobileMarkdown.tsx index 0022daf8d73..31c92d8c9eb 100644 --- a/mobile/src/components/MobileMarkdown.tsx +++ b/mobile/src/components/MobileMarkdown.tsx @@ -13,7 +13,6 @@ import { type ReactNode } from 'react' import { Pressable, ScrollView, Text as NativeText, View, type TextProps } from 'react-native' -import { normalizeMobileMarkdownPreviewHtml } from './mobile-markdown-preview-html' import { styles } from './mobile-markdown-styles' import { detectFilePathSegments, @@ -26,7 +25,8 @@ import { trimAutolinkTrailingPunctuation } from './markdown-inline-token-rules' import { isMobileMermaidLanguage } from './mobile-mermaid-language' -import { parseMobileMarkdown } from './mobile-markdown-parser' +import { useMobileMarkdownBlocks } from './use-mobile-markdown-blocks' +import type { NativeChatVisualDirective } from '../../../src/shared/native-chat-visual-directive' import { MermaidDiagram } from './pr-sidebar/MermaidDiagram' type Props = { @@ -44,6 +44,9 @@ type Props = { * optional :line(:col) suffix). Omitted on screens with no file viewer, where * paths render as plain text (no behavior change). */ onOpenFile?: (pathText: string) => void + /** Native-chat assistant prose only: renders `::orca-visual{...}` directive lines. Without it, + * a directive line is ordinary text. Must be referentially stable (this component is memoized). */ + renderVisual?: (directive: NativeChatVisualDirective, index: number) => ReactNode } const MAX_TABLE_ROWS = 40 @@ -233,14 +236,14 @@ function MobileMarkdownContent({ fallback = '', rangeSelectable = false, textScale = 1, - onOpenFile + onOpenFile, + renderVisual }: Props) { // Interactive children own their touches and must forward the row action. const setup = useContext(MarkdownTextContext) const rowLongPress = setup.androidTranscript ? setup.onLongPress : undefined const text = content?.trim() ?? '' - const previewText = useMemo(() => normalizeMobileMarkdownPreviewHtml(text), [text]) - const blocks = useMemo(() => parseMobileMarkdown(previewText), [previewText]) + const { blocks, directives } = useMobileMarkdownBlocks(text, renderVisual !== undefined) // Scale prose sizes; inline spans inherit fontSize from the wrapping Text. const scaled = (size: number): { fontSize: number; lineHeight: number } | null => textScale !== 1 ? { fontSize: size * textScale, lineHeight: (size + 6) * textScale } : null @@ -259,6 +262,14 @@ function MobileMarkdownContent({ return ( {blocks.map((block, index) => { + if (block.type === 'visual') { + const directive = directives[block.index] + return directive && renderVisual ? ( + + {renderVisual(directive, block.index)} + + ) : null + } if (block.type === 'heading') { return ( ({ + StyleSheet: { create: (styles: T) => styles }, + Text: 'Text', + View: 'View' +})) +vi.mock('./AgentStateDot', () => ({ AgentStateDot: () => null })) +vi.mock('./MobileAgentIcon', () => ({ MobileAgentIcon: () => null })) + +function agent(lastAssistantMessage: string | null): RuntimeWorktreeAgentRow { + return { + paneKey: 'tab:leaf', + parentPaneKey: null, + state: 'done', + agentType: 'claude', + prompt: 'Show usage by day', + taskTitle: null, + displayName: null, + lastAssistantMessage, + toolName: null, + toolInput: null, + interrupted: false, + stateStartedAt: 1_000, + updatedAt: 1_000 + } +} + +describe('WorktreeAgentRow visual lines', () => { + let renderer: ReactTestRenderer | null = null + + afterEach(() => { + act(() => renderer?.unmount()) + renderer = null + }) + + function label(row: RuntimeWorktreeAgentRow): string { + act(() => { + renderer = create( + createElement(WorktreeAgentRow, { agent: row, depth: 0, now: 2_000, unvisited: false }) + ) + }) + return String(renderer!.root.findAll((node) => String(node.type) === 'Text')[0]!.props.children) + } + + it('shows the words around a visual line, never the line itself', () => { + expect(label(agent('Here it is.\n::orca-visual{file="usage.html"}\nSunday dips.'))).toBe( + 'Here it is.\nSunday dips.' + ) + }) + + it('falls back to the prompt when the reply is only a visual', () => { + expect(label(agent('::orca-visual{file="usage.html" title="Usage"}'))).toBe('Show usage by day') + }) +}) diff --git a/mobile/src/components/inline-script-json.ts b/mobile/src/components/inline-script-json.ts new file mode 100644 index 00000000000..cf81bc6d91b --- /dev/null +++ b/mobile/src/components/inline-script-json.ts @@ -0,0 +1,17 @@ +// JSON.stringify escapes quotes and control chars but leaves `<`, `>`, `&`, and +// the U+2028/U+2029 line separators raw — so a value containing `` would +// close the inline ` would -// close the inline ', + mode: 'inline' + }) + + it('puts the visual in an opaque frame that may only run scripts', () => { + const document = build('

hi

') + expect(document).toContain("frame.setAttribute('sandbox', 'allow-scripts')") + expect(document).not.toMatch(/allow-same-origin|allow-popups|allow-top-navigation|allow-forms/) + expect(document.indexOf('Content-Security-Policy')).toBeLessThan(document.indexOf('') + // One script element: the host's own. + expect(document.match(/ + +` +} diff --git a/mobile/src/session/mobile-native-chat-visual-read.test.ts b/mobile/src/session/mobile-native-chat-visual-read.test.ts new file mode 100644 index 00000000000..7ff33904ba0 --- /dev/null +++ b/mobile/src/session/mobile-native-chat-visual-read.test.ts @@ -0,0 +1,200 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import type { RpcClient } from '../transport/rpc-client' +import type { RpcResponse } from '../transport/types' +import { + cachedMobileNativeChatVisual, + readMobileNativeChatVisual, + resetMobileNativeChatVisualCacheForTest, + type MobileNativeChatVisualSource +} from './mobile-native-chat-visual-read' + +const REVISION_A = 'a'.repeat(32) +const REVISION_B = 'b'.repeat(32) + +function success(result: unknown): RpcResponse { + return { id: '1', ok: true, result } +} + +function failure(code: string): RpcResponse { + return { id: '1', ok: false, error: { code, message: code } } +} + +function sourceWith(sendRequest: ReturnType): MobileNativeChatVisualSource { + // oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the reader calls only sendRequest; the rest of RpcClient is connection lifecycle it never touches. + const client = { sendRequest } as unknown as RpcClient + return { client, sessionId: 'session-1' } +} + +afterEach(() => { + resetMobileNativeChatVisualCacheForTest() +}) + +describe('readMobileNativeChatVisual', () => { + it('reads the visual by session and bare file name, then caches it', async () => { + const sendRequest = vi + .fn() + .mockResolvedValue(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' })) + const source = sourceWith(sendRequest) + + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'ready', + html: '

', + revision: REVISION_A + }) + expect(sendRequest).toHaveBeenCalledWith( + 'agentSession.readVisual', + { sessionId: 'session-1', file: 'chart.html' }, + expect.objectContaining({ timeoutMs: expect.any(Number) }) + ) + expect(cachedMobileNativeChatVisual(source, 'chart.html')).toEqual({ + html: '

', + revision: REVISION_A + }) + }) + + it('revalidates with the held revision and keeps the cached bytes on unchanged', async () => { + const sendRequest = vi + .fn() + .mockResolvedValueOnce(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' })) + .mockResolvedValueOnce( + success({ ok: true, revision: REVISION_A, sizeBytes: 3, unchanged: true }) + ) + const source = sourceWith(sendRequest) + await readMobileNativeChatVisual(source, 'chart.html') + + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'ready', + html: '

', + revision: REVISION_A + }) + expect(sendRequest.mock.calls[1]?.[1]).toEqual({ + sessionId: 'session-1', + file: 'chart.html', + knownRevision: REVISION_A + }) + }) + + it('replaces the cached bytes when the host has a new revision', async () => { + const sendRequest = vi + .fn() + .mockResolvedValueOnce(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' })) + .mockResolvedValueOnce( + success({ ok: true, revision: REVISION_B, sizeBytes: 4, html: '


' }) + ) + const source = sourceWith(sendRequest) + await readMobileNativeChatVisual(source, 'chart.html') + await readMobileNativeChatVisual(source, 'chart.html') + expect(cachedMobileNativeChatVisual(source, 'chart.html')?.revision).toBe(REVISION_B) + }) + + it('drops the cached bytes when the host refuses a visual it served before', async () => { + const source = sourceWith( + vi + .fn() + .mockResolvedValueOnce( + success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' }) + ) + .mockResolvedValueOnce(success({ ok: false, error: 'not_found' })) + ) + await readMobileNativeChatVisual(source, 'chart.html') + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'refused' + }) + expect(cachedMobileNativeChatVisual(source, 'chart.html')).toBeNull() + }) + + it('reads a host verdict and an older host as refused', async () => { + for (const response of [ + success({ ok: false, error: 'not_found' }), + success({ ok: false, error: 'some_future_error' }), + failure('method_not_found'), + failure('forbidden') + ]) { + resetMobileNativeChatVisualCacheForTest() + const source = sourceWith(vi.fn().mockResolvedValue(response)) + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'refused' + }) + } + }) + + it('reads an error reply and a malformed reply as no verdict, keeping the cache', async () => { + for (const response of [ + failure('runtime_error'), + success({ ok: true, revision: 'not-hex', html: '' }), + success('') + ]) { + resetMobileNativeChatVisualCacheForTest() + const sendRequest = vi + .fn() + .mockResolvedValueOnce( + success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' }) + ) + .mockResolvedValueOnce(response) + const source = sourceWith(sendRequest) + await readMobileNativeChatVisual(source, 'chart.html') + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'unreachable' + }) + expect(cachedMobileNativeChatVisual(source, 'chart.html')).not.toBeNull() + } + }) + + it('reads a transport failure as unreachable, never as refused', async () => { + const source = sourceWith(vi.fn().mockRejectedValue(new Error('socket closed'))) + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'unreachable' + }) + }) + + it('treats an unchanged answer for a revision it never sent as unreachable and drops the entry', async () => { + const source = sourceWith( + vi.fn().mockResolvedValue(success({ ok: true, revision: REVISION_A, unchanged: true })) + ) + await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({ + kind: 'unreachable' + }) + expect(cachedMobileNativeChatVisual(source, 'chart.html')).toBeNull() + }) + + it('shares one request between concurrent reads of the same visual', async () => { + let resolve: (response: RpcResponse) => void = () => {} + const sendRequest = vi.fn( + () => + new Promise((settle) => { + resolve = settle + }) + ) + const source = sourceWith(sendRequest) + const first = readMobileNativeChatVisual(source, 'chart.html') + const second = readMobileNativeChatVisual(source, 'chart.html') + resolve(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' })) + await expect(Promise.all([first, second])).resolves.toHaveLength(2) + expect(sendRequest).toHaveBeenCalledTimes(1) + }) + + it('keys the cache by client and session, so another host or chat never sees this one', async () => { + const source = sourceWith( + vi + .fn() + .mockResolvedValue(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' })) + ) + await readMobileNativeChatVisual(source, 'chart.html') + expect(cachedMobileNativeChatVisual({ ...source, sessionId: 'other' }, 'chart.html')).toBeNull() + const otherHost = sourceWith(vi.fn()).client + expect(cachedMobileNativeChatVisual({ ...source, client: otherHost }, 'chart.html')).toBeNull() + }) + + it('evicts the oldest visuals past the entry bound', async () => { + const source = sourceWith( + vi + .fn() + .mockResolvedValue(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '

' })) + ) + for (let index = 0; index < 17; index += 1) { + await readMobileNativeChatVisual(source, `v${index}.html`) + } + expect(cachedMobileNativeChatVisual(source, 'v0.html')).toBeNull() + expect(cachedMobileNativeChatVisual(source, 'v16.html')).not.toBeNull() + }) +}) diff --git a/mobile/src/session/mobile-native-chat-visual-read.ts b/mobile/src/session/mobile-native-chat-visual-read.ts new file mode 100644 index 00000000000..7b46618a4f4 --- /dev/null +++ b/mobile/src/session/mobile-native-chat-visual-read.ts @@ -0,0 +1,169 @@ +import { z } from 'zod' +import { NATIVE_CHAT_VISUAL_MAX_BYTES } from '../../../src/shared/native-chat-visual-directive' +import { bindDeferredRpcOperation, defineRpcOperation } from '../transport/rpc-operation' +import { rpcResultVariant } from '../transport/rpc-operation-result-reader' +import { isMobileMethodUnavailableError } from '../transport/mobile-method-unavailable' +import type { RpcClient } from '../transport/rpc-client' + +/** Where a chat's visuals are read from: the host that owns the structured session. */ +export type MobileNativeChatVisualSource = { + client: RpcClient + sessionId: string +} + +export type MobileNativeChatVisualRead = + | { kind: 'ready'; html: string; revision: string } + /** The host positively refused (missing, too large, outside the folder...) or cannot serve visuals. */ + | { kind: 'refused' } + /** No verdict: no answer, an error reply, an unreadable answer. Worth asking again. */ + | { kind: 'unreachable' } + +const Revision = z.string().regex(/^[0-9a-f]{16,64}$/) + +const visualReadReplySchema = z.union([ + z.object({ + ok: z.literal(true), + revision: Revision, + html: z.string().max(NATIVE_CHAT_VISUAL_MAX_BYTES) + }), + z.object({ ok: z.literal(true), revision: Revision, unchanged: z.literal(true) }), + // A newer host's error code still reads as a refusal. + z.object({ ok: z.literal(false), error: z.string() }) +]) + +/** An error reply and an unreadable reply both read as null. */ +const nativeChatVisualRead = bindDeferredRpcOperation( + defineRpcOperation({ + name: 'agentSession.read-visual', + method: 'agentSession.readVisual', + acceptance: 'object-result-or-null', + barrier: 'after-caller-barrier', + read: rpcResultVariant('native-chat-visual', visualReadReplySchema) + }) +) + +const READ_TIMEOUT_MS = 15_000 +// Each entry is at most 512 KiB of UTF-8; the character bound is what caps memory. +const MAX_CACHE_ENTRIES = 16 +const MAX_CACHE_CHARS = 4 * 1024 * 1024 + +type CachedVisual = { html: string; revision: string } + +const cache = new Map() +const inFlight = new Map>() + +// The client stands for the host: each paired host has its own, so one host's visuals never +// answer for another's even if two session ids collided. +const clientKeys = new WeakMap() +let nextClientKey = 0 + +function cacheKey(source: MobileNativeChatVisualSource, file: string): string { + let clientKey = clientKeys.get(source.client) + if (clientKey === undefined) { + nextClientKey += 1 + clientKey = nextClientKey + clientKeys.set(source.client, clientKey) + } + return JSON.stringify([clientKey, source.sessionId, file]) +} + +function remember(key: string, entry: CachedVisual): void { + cache.delete(key) + cache.set(key, entry) + let chars = 0 + for (const value of cache.values()) { + chars += value.html.length + } + // Oldest first: Map iteration follows insertion order, and a hit re-inserts. + for (const [oldKey, value] of cache) { + if (cache.size <= MAX_CACHE_ENTRIES && chars <= MAX_CACHE_CHARS) { + break + } + if (oldKey === key) { + continue + } + cache.delete(oldKey) + chars -= value.html.length + } +} + +/** What this phone already holds for a visual, to paint before revalidating. */ +export function cachedMobileNativeChatVisual( + source: MobileNativeChatVisualSource, + file: string +): CachedVisual | null { + return cache.get(cacheKey(source, file)) ?? null +} + +async function readOnce( + source: MobileNativeChatVisualSource, + file: string, + key: string +): Promise { + const known = cache.get(key) + let response + try { + response = await nativeChatVisualRead.request( + source.client, + { + sessionId: source.sessionId, + file, + ...(known ? { knownRevision: known.revision } : {}) + }, + { timeoutMs: READ_TIMEOUT_MS, budgetSpansConnect: true } + ) + } catch { + return { kind: 'unreachable' } + } + if (!response.ok && isMobileMethodUnavailableError(response.error.code, response.error.message)) { + // An older host (its mobile allowlist answers `forbidden`): asking again will not help. + return { kind: 'refused' } + } + const reply = nativeChatVisualRead.interpret(response) + if (reply === null) { + // An error reply, an older host or an unreadable answer is not a verdict on the file. + return { kind: 'unreachable' } + } + if (!reply.ok) { + // The host no longer serves it (deleted, now too large...): stop painting the old bytes. + cache.delete(key) + return { kind: 'refused' } + } + if ('html' in reply) { + remember(key, { html: reply.html, revision: reply.revision }) + return { kind: 'ready', html: reply.html, revision: reply.revision } + } + // `unchanged` answers the revision this phone sent; anything else is a host that lost track. + if (known && known.revision === reply.revision) { + remember(key, known) + return { kind: 'ready', html: known.html, revision: known.revision } + } + cache.delete(key) + return { kind: 'unreachable' } +} + +/** + * Reads one visual from the chat's owning host, revalidating what the phone already holds by + * revision. Concurrent reads of the same visual share one request. + */ +export function readMobileNativeChatVisual( + source: MobileNativeChatVisualSource, + file: string +): Promise { + const key = cacheKey(source, file) + const pending = inFlight.get(key) + if (pending) { + return pending + } + const read = readOnce(source, file, key).finally(() => { + inFlight.delete(key) + }) + inFlight.set(key, read) + return read +} + +/** Test seam: forget every cached visual. */ +export function resetMobileNativeChatVisualCacheForTest(): void { + cache.clear() + inFlight.clear() +} diff --git a/mobile/src/session/mobile-native-chat-visual-theme.ts b/mobile/src/session/mobile-native-chat-visual-theme.ts new file mode 100644 index 00000000000..08f244887d5 --- /dev/null +++ b/mobile/src/session/mobile-native-chat-visual-theme.ts @@ -0,0 +1,40 @@ +import type { NativeChatVisualTheme } from '../../../src/shared/native-chat-visual-shell' +import { colors, radii } from '../theme/mobile-theme' + +/** + * The phone's one (dark) theme, as the variables a visual styles against. The app has no theme + * switch, so a visual receives this before its content runs and never needs a live update. + * Chart colors are desktop's dark `--chart-1..5` (the blue 300-800 ramp), as hex. + */ +export const MOBILE_NATIVE_CHAT_VISUAL_THEME: NativeChatVisualTheme = { + colorScheme: 'dark', + tokens: { + '--background': colors.bgBase, + '--foreground': colors.textPrimary, + '--card': colors.bgPanel, + '--card-foreground': colors.textPrimary, + '--popover': colors.bgRaised, + '--popover-foreground': colors.textPrimary, + '--primary': colors.surfaceBright, + '--primary-foreground': colors.bgBase, + '--secondary': colors.bgRaised, + '--secondary-foreground': colors.textPrimary, + '--muted': colors.bgRaised, + '--muted-foreground': colors.textMuted, + '--accent': colors.bgRaised, + '--accent-foreground': colors.textPrimary, + '--destructive': colors.statusRed, + '--destructive-foreground': colors.onAccent, + '--border': colors.borderSubtle, + '--input': colors.borderSubtle, + '--ring': colors.textMuted, + '--radius': `${radii.row}px`, + '--chart-1': '#8ec5ff', + '--chart-2': '#2b7fff', + '--chart-3': '#155dfc', + '--chart-4': '#1447e6', + '--chart-5': '#193cb8', + '--font-sans': '-apple-system, system-ui, Roboto, sans-serif', + '--font-mono': 'Menlo, monospace' + } +} diff --git a/mobile/src/session/use-mobile-native-chat-controller.ts b/mobile/src/session/use-mobile-native-chat-controller.ts index c44001a4573..3a57cfb2aff 100644 --- a/mobile/src/session/use-mobile-native-chat-controller.ts +++ b/mobile/src/session/use-mobile-native-chat-controller.ts @@ -298,6 +298,7 @@ export function useMobileNativeChatController(args: { nativeChatSession, /** Structured lane: drives the per-turn status row and live tool progress. */ nativeChatStructured: activeChatStructured, + nativeChatVisualSource: activeChatStructured ? structuredNativeChat.visualSource : null, nativeChatAgentWorking, nativeChatTurnIndicator: activeChatStructured ? structuredNativeChat.turnIndicator : null, nativeChatWorkingStartedAt: activeChatStructured ? structuredNativeChat.workingStartedAt : null, diff --git a/mobile/src/session/use-mobile-native-chat-turn-disclosure-growing-row.test.tsx b/mobile/src/session/use-mobile-native-chat-turn-disclosure-growing-row.test.tsx new file mode 100644 index 00000000000..90fda385bcc --- /dev/null +++ b/mobile/src/session/use-mobile-native-chat-turn-disclosure-growing-row.test.tsx @@ -0,0 +1,49 @@ +import { createElement } from 'react' +import { act, create, type ReactTestRenderer } from 'react-test-renderer' +import { afterEach, describe, expect, it } from 'vitest' +import type { NativeChatMessage } from '../../../src/shared/native-chat-types' +import { Harness, Result, userMessage } from './use-mobile-native-chat-turn-disclosure.test-fixture' + +function assistantMessage(id: string): NativeChatMessage { + return { + id, + role: 'assistant', + blocks: [{ type: 'text', text: id }], + timestamp: null, + source: 'transcript' + } +} + +describe('useMobileNativeChatTurnDisclosure growing row', () => { + let renderer: ReactTestRenderer | null = null + + afterEach(() => { + act(() => renderer?.unmount()) + renderer = null + }) + + it('marks no row while a prompt is open', () => { + const messages = [userMessage('u1'), assistantMessage('a1')] + act(() => { + renderer = create(createElement(Harness, { messages, enabled: true, lineYields: true })) + }) + const { disclosure } = renderer!.root.findByType(Result).props + expect(disclosure.resolveRow(1, messages[1]).mayStillGrow).toBe(false) + }) + + it('marks only the newest assistant row, even when a user row follows it', () => { + const messages = [ + userMessage('u1'), + assistantMessage('a1'), + assistantMessage('a2'), + userMessage('u2') + ] + act(() => { + renderer = create(createElement(Harness, { messages, enabled: true })) + }) + const { disclosure } = renderer!.root.findByType(Result).props + expect( + messages.map((message, index) => disclosure.resolveRow(index, message).mayStillGrow) + ).toEqual([false, false, true, false]) + }) +}) diff --git a/mobile/src/session/use-mobile-native-chat-turn-disclosure.ts b/mobile/src/session/use-mobile-native-chat-turn-disclosure.ts index 28bd7a7d4fa..300a792191f 100644 --- a/mobile/src/session/use-mobile-native-chat-turn-disclosure.ts +++ b/mobile/src/session/use-mobile-native-chat-turn-disclosure.ts @@ -33,6 +33,8 @@ export type MobileNativeChatTurnRow = { /** Set only on a settled turn — the one row that has activity to disclose. */ turnKey?: string activeTurnIsWorking: boolean + /** The newest assistant row of a live turn with no prompt open: its last text may still grow. */ + mayStillGrow: boolean /** The live activity line discloses this open reasoning block, so its row draws nothing. */ reasoningIsLive: boolean /** A reasoning row's disclosure, keyed like the live line's so an opened block stays open. */ @@ -195,6 +197,10 @@ export function useMobileNativeChatTurnDisclosure({ }, [expandedReasoning, line] ) + const latestAssistantId = useMemo( + () => waiting.listMessages.findLast((row) => row.role === 'assistant')?.id ?? null, + [waiting.listMessages] + ) const resolveRow = useCallback( (listIndex: number, message: NativeChatMessage): MobileNativeChatTurnRow => { const index = waiting.indexById?.get(message.id) ?? listIndex @@ -219,6 +225,7 @@ export function useMobileNativeChatTurnDisclosure({ turnKey: turnKey && turnStatus?.workedSeconds != null ? turnKey : undefined, // With no user boundary at all, the session's working state stays authoritative. activeTurnIsWorking: inLiveWorkingTurn(index), + mayStillGrow: !lineYields && message.id === latestAssistantId, reasoningIsLive: message.id === liveReasoningId, reasoningExpanded: message.role === 'reasoning' && @@ -236,6 +243,8 @@ export function useMobileNativeChatTurnDisclosure({ completedByTurn, expandedTurnIds, inLiveWorkingTurn, + latestAssistantId, + lineYields, liveReasoningId, expandedReasoning, toggleReasoning diff --git a/mobile/src/session/use-mobile-native-chat-visual.test.tsx b/mobile/src/session/use-mobile-native-chat-visual.test.tsx new file mode 100644 index 00000000000..21e8cde258c --- /dev/null +++ b/mobile/src/session/use-mobile-native-chat-visual.test.tsx @@ -0,0 +1,92 @@ +import { createElement } from 'react' +import { act, create, type ReactTestRenderer } from 'react-test-renderer' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import type { RpcClient } from '../transport/rpc-client' +import type { MobileNativeChatVisualRead } from './mobile-native-chat-visual-read' + +const reads = vi.hoisted(() => { + const next: MobileNativeChatVisualRead[] = [] + return { next, calls: 0 } +}) + +vi.mock('./mobile-native-chat-visual-read', () => ({ + cachedMobileNativeChatVisual: () => null, + readMobileNativeChatVisual: () => { + reads.calls += 1 + return Promise.resolve(reads.next.shift() ?? { kind: 'unreachable' }) + } +})) + +import { + useMobileNativeChatVisual, + type MobileNativeChatVisualState +} from './use-mobile-native-chat-visual' + +// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the read module is mocked; the client is only an identity here. +const source = { client: {} as RpcClient, sessionId: 'session-1' } + +let latest: { state: MobileNativeChatVisualState; retry: () => void } | null = null +function Probe() { + latest = useMobileNativeChatVisual(source, 'chart.html') + return null +} + +describe('useMobileNativeChatVisual', () => { + let renderer: ReactTestRenderer | null = null + + beforeEach(() => { + vi.useFakeTimers() + reads.next = [] + reads.calls = 0 + }) + + afterEach(() => { + act(() => renderer?.unmount()) + renderer = null + latest = null + vi.useRealTimers() + }) + + async function mount(): Promise { + await act(async () => { + renderer = create(createElement(Probe)) + }) + } + + it('re-asks a bounded number of times when the host does not answer, then shows unavailable', async () => { + await mount() + expect(reads.calls).toBe(1) + for (const delay of [1_500, 5_000]) { + await act(async () => { + await vi.advanceTimersByTimeAsync(delay) + }) + } + expect(reads.calls).toBe(3) + expect(latest?.state).toEqual({ kind: 'unavailable' }) + await act(async () => { + await vi.advanceTimersByTimeAsync(60_000) + }) + expect(reads.calls).toBe(3) + }) + + it('does not latch a failure: a retry asks again and can succeed', async () => { + reads.next = [{ kind: 'refused' }, { kind: 'ready', html: '

', revision: 'a'.repeat(32) }] + await mount() + expect(latest?.state).toEqual({ kind: 'unavailable' }) + await act(async () => { + latest?.retry() + }) + expect(reads.calls).toBe(2) + expect(latest?.state).toEqual({ kind: 'ready', html: '

', revision: 'a'.repeat(32) }) + }) + + it('stops re-asking after unmount', async () => { + await mount() + act(() => renderer?.unmount()) + renderer = null + await act(async () => { + await vi.advanceTimersByTimeAsync(60_000) + }) + expect(reads.calls).toBe(1) + }) +}) diff --git a/mobile/src/session/use-mobile-native-chat-visual.ts b/mobile/src/session/use-mobile-native-chat-visual.ts new file mode 100644 index 00000000000..cb06138d5d0 --- /dev/null +++ b/mobile/src/session/use-mobile-native-chat-visual.ts @@ -0,0 +1,87 @@ +import { useCallback, useEffect, useState } from 'react' +import { + cachedMobileNativeChatVisual, + readMobileNativeChatVisual, + type MobileNativeChatVisualSource +} from './mobile-native-chat-visual-read' + +export type MobileNativeChatVisualState = + | { kind: 'loading' } + | { kind: 'ready'; html: string; revision: string } + | { kind: 'unavailable' } + +/** Automatic re-asks after a read that got no answer; the reader's tap starts a fresh round. */ +const AUTO_RETRY_DELAYS_MS = [1_500, 5_000] as const + +function initialState( + source: MobileNativeChatVisualSource, + file: string +): MobileNativeChatVisualState { + const cached = cachedMobileNativeChatVisual(source, file) + return cached ? { kind: 'ready', ...cached } : { kind: 'loading' } +} + +/** + * One visual's content: painted from the phone's cache when it has one, revalidated against the + * owning host on every mount, and retried a bounded number of times when the host does not answer. + * Nothing latches: `retry` (or a new client after a reconnect) asks again. + */ +export function useMobileNativeChatVisual( + source: MobileNativeChatVisualSource, + file: string +): { state: MobileNativeChatVisualState; retry: () => void } { + const [state, setState] = useState(() => initialState(source, file)) + // `round` is a reader's retry; `retriesUsed` counts automatic re-asks within it. + const [request, setRequest] = useState({ round: 0, retriesUsed: 0 }) + const [retryInMs, setRetryInMs] = useState(null) + + useEffect(() => { + let disposed = false + void readMobileNativeChatVisual(source, file).then((read) => { + if (disposed) { + return + } + if (read.kind === 'ready') { + // Same revision keeps the same state object, so the frame does not reload its document. + setState((current) => + current.kind === 'ready' && current.revision === read.revision + ? current + : { kind: 'ready', html: read.html, revision: read.revision } + ) + return + } + const delay = + read.kind === 'unreachable' ? AUTO_RETRY_DELAYS_MS[request.retriesUsed] : undefined + if (delay !== undefined) { + setRetryInMs(delay) + return + } + // A host that does not answer leaves a visual on screen; one that refuses it takes it down. + setState((current) => + current.kind === 'ready' && read.kind === 'unreachable' ? current : { kind: 'unavailable' } + ) + }) + return () => { + disposed = true + } + }, [source, file, request]) + + useEffect(() => { + if (retryInMs === null) { + return + } + const timer = setTimeout(() => { + setRetryInMs(null) + setRequest((current) => ({ ...current, retriesUsed: current.retriesUsed + 1 })) + }, retryInMs) + return () => clearTimeout(timer) + }, [retryInMs]) + + const retry = useCallback(() => { + setState((current) => (current.kind === 'unavailable' ? { kind: 'loading' } : current)) + setRetryInMs(null) + setRequest((current) => ({ round: current.round + 1, retriesUsed: 0 })) + }, []) + + return { state, retry } +} diff --git a/mobile/src/session/use-mobile-structured-agent-session.ts b/mobile/src/session/use-mobile-structured-agent-session.ts index 6bcc502aaf2..02226b2adcd 100644 --- a/mobile/src/session/use-mobile-structured-agent-session.ts +++ b/mobile/src/session/use-mobile-structured-agent-session.ts @@ -18,6 +18,7 @@ import { projectStructuredQuestion } from './mobile-structured-agent-prompts' import type { RpcClient } from '../transport/rpc-client' +import type { MobileNativeChatVisualSource } from './mobile-native-chat-visual-read' import type { MobileChatPermission } from './mobile-native-chat-permission' import type { MobileChatQuestion } from './mobile-native-chat-question' import type { MobileNativeChatSession } from './use-mobile-native-chat-session' @@ -67,6 +68,8 @@ type StructuredMobileSession = ReturnType Promise /** The queued-draft cards and their actions, from any host that publishes them. */ queued: MobileStructuredQueuedMessageControls + /** Where this chat's `::orca-visual` lines read their HTML from; null without a client. */ + visualSource: MobileNativeChatVisualSource | null } export function useMobileStructuredAgentSession(args: { @@ -266,8 +269,14 @@ export function useMobileStructuredAgentSession(args: { ] ) + const visualSource = useMemo( + () => (client && sessionId ? { client, sessionId } : null), + [client, sessionId] + ) + return { ...options, + visualSource, session: { messages, status, diff --git a/mobile/web-entry/web-overrides.json b/mobile/web-entry/web-overrides.json index c1ba2a2a92e..8c917694ce5 100644 --- a/mobile/web-entry/web-overrides.json +++ b/mobile/web-entry/web-overrides.json @@ -161,6 +161,10 @@ "file": "src/components/AuthFailedBannerActions.web.tsx", "reason": "The auth-failed banner offers Retry, Re-pair and Remove, and on the page only Re-pair works. /pair-scan sits outside the page's route root of app/h (config/scripts/mobile-web-app-route-manifest.mjs:6), so route-handoff.web.ts:207 hands the push to the shell, which opens the native scan screen and returns to the same page document on Back. Retry and Remove are inert here: forceReconnect is null (client-context.web.tsx:57, read through host-client-hooks.ts:88) and removal refuses (page-host-removal-refusal.ts). The banner is reachable on the page because the shell forwards the native client's state (mobile-web-shell/bridge-host.ts:301) and auth-failed is in the wire enum (bridge/bridge-envelope.ts:44). So this one renders Re-pair as native does and one line naming the app for reconnect and removal." }, + { + "file": "src/session/MobileNativeChatVisualFrame.web.tsx", + "reason": "The native frame runs a chat visual's scripts in a sandboxed child of a trusted host document inside a react-native-webview, which throws at import in a browser. The page's response-header policy (script-src 'self', frame-src 'none') is inherited by a srcdoc frame, so a visual's inline scripts cannot run here without loosening the page policy. This one renders the visual sealed, as MobileHtmlPreview.web.tsx does: an empty sandbox, no script, a fixed-height frame the visual scrolls inside." + }, { "file": "src/components/MobileHtmlPreview.web.tsx", "reason": "The native preview renders an agent-produced HTML artifact inside a sandboxed WebView with navigation locked to the initial inline document, and react-native-webview throws at import in a browser for the reason above. This one renders the artifact in a sandboxed iframe with no allow-scripts and no allow-same-origin, keeping the Preview/Source toggle. srcdoc rather than a blob: URL and no CSP change at all: a srcdoc frame has no URL for frame-src to match and inherits its embedder's policy instead, so it is admitted under the shipped frame-src 'none' on Chromium and WebKit alike, while a blob: frame is refused by frame-src and refused again in WebKit by the frame-ancestors 'none' it inherits. The inherited policy is also what seals it -- script-src 'self' refuses the artifact's inline script, img-src bounds its images, font-src 'none' its fonts -- and allow-top-navigation-by-user-activation is the one capability granted, so a tapped link becomes a top-frame navigation the shell opens externally (ruling 29) while a meta refresh, a form submit, target=_blank and any script-initiated navigation produce none. That token is granted only when the shell says it can open such a navigation: without the externalNavigation grant this file renders the artifact with its links as text and seals the frame with an empty sandbox instead (C8.1, ruling 37.2)." diff --git a/src/main/runtime/mobile-rpc-allowlist.test.ts b/src/main/runtime/mobile-rpc-allowlist.test.ts index 1842d0a8421..e761fd9e04c 100644 --- a/src/main/runtime/mobile-rpc-allowlist.test.ts +++ b/src/main/runtime/mobile-rpc-allowlist.test.ts @@ -170,6 +170,7 @@ describe('mobile RPC allowlist', () => { 'agentSession.subscribe', 'agentSession.unsubscribe', 'agentSession.subscribeStatus', + 'agentSession.readVisual', 'agentSession.hold', 'agentSession.release' ]) diff --git a/src/main/runtime/runtime-rpc/runtime-rpc-mobile-agent-session-methods.ts b/src/main/runtime/runtime-rpc/runtime-rpc-mobile-agent-session-methods.ts index 53c0a871703..56b30d837a5 100644 --- a/src/main/runtime/runtime-rpc/runtime-rpc-mobile-agent-session-methods.ts +++ b/src/main/runtime/runtime-rpc/runtime-rpc-mobile-agent-session-methods.ts @@ -23,6 +23,8 @@ export const MOBILE_AGENT_SESSION_RPC_METHODS = [ 'agentSession.unsubscribe', // Every session's status on one stream: the phone's chat reads the host's "Stopping…" from it. 'agentSession.subscribeStatus', + // One visual from the chat's own visuals folder, for the transcript's `::orca-visual` lines. + 'agentSession.readVisual', // No-ops on a current host; kept until MIN_COMPATIBLE_RUNTIME_CLIENT_VERSION passes the // mobile builds that still call them. 'agentSession.hold',