mirror of
https://github.com/stablyai/orca.git
synced 2026-10-09 08:02:35 +00:00
feat(mobile): show chat visuals inline in the phone's native chat (#26071)
* feat(native-chat): visual directive grammar and host read for a chat's visuals folder
A shared grammar for the ::orca-visual{file="..." title="..."} reply line,
the per-chat visuals folder location on the owning host, and the
agentSession.readVisual runtime method that reads one visual with lexical and
canonical containment, a 512 KiB bounded read and UTF-8 refusal.
* feat(native-chat): shared frame document for chat visuals
One string builder every client wraps a visual's HTML with: the policy
(CDN assets only, no fetch, frames, workers, forms or base rewrites),
the theme variables, and a prelude that reports height, routes links to
the parent and refuses navigation. Also the validated frame-to-parent
message reader and the live theme message.
* feat(mobile): render native-chat visuals inline in the phone's chat
A finished assistant reply's ::orca-visual line now shows the visual
inline, read from the chat's owning host through agentSession.readVisual.
The visual runs in an opaque sandboxed child of a trusted host document
inside the WebView; the app accepts only a token-checked height and an
http(s) link opened under user activation. Navigation away from the
visual's own document is refused, a dead web process reloads once, and
the visual opens full screen. The hybrid shell's page renders it sealed.
* feat(native-chat): render chat visuals inline and in the right sidebar
Native-chat assistant replies render a ::orca-visual{...} line as the chat's
HTML visual in an opaque, scripts-only sandboxed frame: CSP first, the host
frame navigation guard registered before content runs, live theme without a
reload, fitted height, links opened in the viewer's browser only from a real
gesture, lazy mount, and one muted line when the visual cannot be shown.
Open in sidebar shows the same frame in the right sidebar, widened while it
is open and restored after.
* fix(mobile): chat visuals use the shared shell; only the host page may message the app
Review round 1:
- use PR 1's shared visual shell and height governor instead of a second
builder; the app decides heights and pushes them to the host page
- react-native-webview patch: the message channel accepts only string
messages from the main frame on iOS and Android, so a visual in its
sandboxed child cannot reach it (or crash Android with a non-string)
- structured replies grow in place: while a turn works, a row holds back a
directive still being typed at its tail; finished lines mount
- links need child focus + activation, one per activation window
- a refused read takes the visual down and drops its cached bytes
- in-page anchors load; text spelling a placeholder renders no visuals
* refactor(native-chat): move the visual height governor to src/shared
The phone bundles only src/shared, so the governor the desktop frame uses
moves there unchanged and the mobile frame shares it.
* fix(native-chat): point the desktop and mobile frames at the moved governor
* fix(mobile): review round 2 for chat visuals
- host page relays only messages on the visual's own channel, as its own
copy, so a visual cannot push oversized fields through it; relay rate
halved; a link is validated before it uses up the link window
- the frame denies camera, microphone, geolocation, clipboard and display
capture; iOS media capture requests are denied
- react-native-webview patch: the iOS history-shim handler also accepts
only main-frame string messages
- only the newest assistant row of a working turn holds back an unfinished
directive; earlier finished rows show their visuals
- an error reply or older host is not a verdict: it keeps a visual on
screen and its cache, and retries; only a host refusal takes it down
* fix(mobile): a drag that starts on an inline visual scrolls the chat
iOS gives a scrollable frame its own scroll view, which took the drag;
the inline frame is sized to its content, so it no longer scrolls.
Full screen still does.
* fix(mobile): review round 3 for chat visuals
- hold back only the last block of the newest assistant row, and not
while a question or approval is open, so a visual followed by a tool
call or a pending question shows at once
- an older host (method_not_found) reads as unavailable without retries
- drop a stray @pnpm/exe lockfile block; only the patch hash changes
* test(mobile): list the visual frame's web sibling; mock it in the prompt-controller harness
* fix(mobile): an older desktop's mobile allowlist refusal reads as unavailable at once
* fix(native-chat): visual CI fixes, shared height governor, live-turn streaming hold
Registers agentSession.readVisual from the methods index so the structured
method file stays under its line budget, replaces reflective reads with checked
narrowing, moves the pure height governor to src/shared for mobile, and holds a
half-written directive tail while the turn works (structured text rows carry no
running state).
* fix(native-chat): harden the visual read and link opening
Re-checks after the open that the chat's visuals folder is still the real
directory at Orca's path, reports unexpected filesystem faults by code without
host paths, and lets one click in a visual open at most one page.
* fix(native-chat): keep visual lines out of plain-text reply surfaces; review fixes
One shared helper drops visual lines (outside fenced code) from reply text where
it becomes plain text: the structured status summary that feeds the sidebar row,
dashboard, notifications, phone rows and handoffs, and AI Vault reply previews.
Review fixes: height also counts a pinned body's overflow, only the live
frontier row holds a half-written visual line, the runaway-height stop needs the
same step repeated, and any host refusal evicts the cached revision.
* fix(native-chat): resolve the visuals folder without the removed journal-paths helper
Main removed the per-chat journal paths and the journal database's state
directory; the visuals folder keeps the same sha256 layout on its own and the
read method uses the profile state directory the chat host is opened in.
* fix(mobile): keep visual lines out of the worktree row and copied message text
A reply's ::orca-visual line renders only in the transcript. The
worktree list's agent row and the message actions sheet's copy text now
drop it with the shared helper; a reply that is only a visual falls back
to the prompt, as an empty one does.
* fix(native-chat): review round 2 fixes; copy a reply without its visual lines
Reply previews in Agent Session History drop visual lines per text part before
lines are folded; the frame adds a body's overflow only when the body really
overflows; fence tracking follows CommonMark closers and openers; the copy
button copies a reply without visual lines; a coded read fault keeps its cause.
* fix(native-chat): update the frame's theme ref after render; read the visuals folder pair at once
* test(native-chat): declare agentSession.readVisual on the cross-version agent-session surface
* fix(native-chat): copying a reply keeps its code blocks and indentation
Removing visual lines now closes only the gap each removal leaves, instead of
collapsing blank lines across the whole reply and trimming its indentation; the
visuals folder is checked parent first again so a broken path answers the same
way every time.
This commit is contained in:
@@ -1,3 +1,46 @@
|
||||
diff --git a/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java b/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java
|
||||
index 80c68034b193a47bedb56ca527352de9548e8ac6..977c068a74c88c0322fca2a4bbffab8bc1ba719d 100644
|
||||
--- a/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java
|
||||
+++ b/android/src/main/java/com/reactnativecommunity/webview/RNCWebView.java
|
||||
@@ -253,6 +253,12 @@ public class RNCWebView extends WebView implements LifecycleEventListener {
|
||||
this.bridgeListener = new WebViewCompat.WebMessageListener() {
|
||||
@Override
|
||||
public void onPostMessage(@NonNull WebView view, @NonNull WebMessageCompat message, @NonNull Uri sourceOrigin, boolean isMainFrame, @NonNull JavaScriptReplyProxy replyProxy) {
|
||||
+ // Orca: only the page's own document may message the app. A subframe (for example agent
|
||||
+ // HTML in a sandboxed iframe) reaches this listener too, and getData() throws for a
|
||||
+ // non-string message.
|
||||
+ if (!isMainFrame || message.getType() != WebMessageCompat.TYPE_STRING) {
|
||||
+ return;
|
||||
+ }
|
||||
RNCWebView.this.onMessage(message.getData(), sourceOrigin.toString());
|
||||
}
|
||||
};
|
||||
diff --git a/apple/RNCWebViewImpl.m b/apple/RNCWebViewImpl.m
|
||||
index 7f5c24d6e2cfb26eddbc513bf17a81c24a7273e2..d70cb58a3c81cb22bd549a2c37340ba53853cfcf 100644
|
||||
--- a/apple/RNCWebViewImpl.m
|
||||
+++ b/apple/RNCWebViewImpl.m
|
||||
@@ -776,12 +776,21 @@ - (void)userContentController:(WKUserContentController *)userContentController
|
||||
didReceiveScriptMessage:(WKScriptMessage *)message
|
||||
{
|
||||
if ([message.name isEqualToString:HistoryShimName]) {
|
||||
+ // Orca: the history shim is the page's own script; a subframe or non-string body is not it.
|
||||
+ if (!message.frameInfo.isMainFrame || ![message.body isKindOfClass:[NSString class]]) {
|
||||
+ return;
|
||||
+ }
|
||||
if (_onLoadingFinish) {
|
||||
NSMutableDictionary<NSString *, id> *event = [self baseEvent];
|
||||
[event addEntriesFromDictionary: @{@"navigationType": message.body}];
|
||||
_onLoadingFinish(event);
|
||||
}
|
||||
} else if ([message.name isEqualToString:MessageHandlerName]) {
|
||||
+ // Orca: only the page's own document may message the app. A subframe (for example agent HTML in
|
||||
+ // a sandboxed iframe) reaches this handler too, and a non-string body is not a message.
|
||||
+ if (!message.frameInfo.isMainFrame || ![message.body isKindOfClass:[NSString class]]) {
|
||||
+ return;
|
||||
+ }
|
||||
if (_onMessage) {
|
||||
NSMutableDictionary<NSString *, id> *event = [self baseEvent];
|
||||
[event addEntriesFromDictionary: @{@"data": message.body}];
|
||||
diff --git a/index.d.ts b/index.d.ts
|
||||
index 10b9c145b920a6a2ba1fb09bd72150328fafe211..626a67a3d6aca587ee9e74a52b799a2c50b4c52b 100644
|
||||
--- a/index.d.ts
|
||||
|
||||
Generated
+4
-4
@@ -172,7 +172,7 @@ overrides:
|
||||
patchedDependencies:
|
||||
expo-notifications@55.0.27: ce20843a3daad4185d7e8571788fa323ba4d11984936188790858650a61749c0
|
||||
query-string@9.5.1: 8624f1beedd0705bfdaf287519ed6363662a0c96f01fd594508b34583bf30d8b
|
||||
react-native-webview@13.16.2: de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27
|
||||
react-native-webview@13.16.2: 55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06
|
||||
react-native@0.83.10: 44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d
|
||||
|
||||
importers:
|
||||
@@ -310,7 +310,7 @@ importers:
|
||||
version: 0.21.3(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
|
||||
react-native-webview:
|
||||
specifier: 13.16.2
|
||||
version: 13.16.2(patch_hash=de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
version: 13.16.2(patch_hash=55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
react-native-worklets:
|
||||
specifier: ^0.8.3
|
||||
version: 0.8.3(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
@@ -12228,7 +12228,7 @@ snapshots:
|
||||
optionalDependencies:
|
||||
'@expo/dom-webview': 55.0.6(expo@55.0.31)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
'@expo/metro-runtime': 55.0.12(@expo/dom-webview@55.0.6)(expo@55.0.31)(react-dom@19.2.8(react@19.2.8))(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
react-native-webview: 13.16.2(patch_hash=de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
react-native-webview: 13.16.2(patch_hash=55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8)
|
||||
transitivePeerDependencies:
|
||||
- '@babel/core'
|
||||
- bufferutil
|
||||
@@ -14343,7 +14343,7 @@ snapshots:
|
||||
transitivePeerDependencies:
|
||||
- encoding
|
||||
|
||||
react-native-webview@13.16.2(patch_hash=de6761dfa76a5491a23e49f1262566a8831cab26736a336fdffc5d4e7d05ff27)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8):
|
||||
react-native-webview@13.16.2(patch_hash=55b7bce35c78ee1a50c4b99ad65b52982f3cfa73da6f61d0853c4d03219eec06)(react-native@0.83.10(patch_hash=44876634a8efbb0f2c3f66cd4332be170ec821d1cbfc0264ac80680983e8513d)(@babel/core@7.29.7)(@react-native/metro-config@0.85.2(@babel/core@7.29.7))(@types/react@19.2.18)(react@19.2.8))(react@19.2.8):
|
||||
dependencies:
|
||||
'@typescript/native-preview': 7.0.0-dev.20260707.2
|
||||
escape-string-regexp: 4.0.0
|
||||
|
||||
@@ -13,7 +13,6 @@ import {
|
||||
type ReactNode
|
||||
} from 'react'
|
||||
import { Pressable, ScrollView, Text as NativeText, View, type TextProps } from 'react-native'
|
||||
import { normalizeMobileMarkdownPreviewHtml } from './mobile-markdown-preview-html'
|
||||
import { styles } from './mobile-markdown-styles'
|
||||
import {
|
||||
detectFilePathSegments,
|
||||
@@ -26,7 +25,8 @@ import {
|
||||
trimAutolinkTrailingPunctuation
|
||||
} from './markdown-inline-token-rules'
|
||||
import { isMobileMermaidLanguage } from './mobile-mermaid-language'
|
||||
import { parseMobileMarkdown } from './mobile-markdown-parser'
|
||||
import { useMobileMarkdownBlocks } from './use-mobile-markdown-blocks'
|
||||
import type { NativeChatVisualDirective } from '../../../src/shared/native-chat-visual-directive'
|
||||
import { MermaidDiagram } from './pr-sidebar/MermaidDiagram'
|
||||
|
||||
type Props = {
|
||||
@@ -44,6 +44,9 @@ type Props = {
|
||||
* optional :line(:col) suffix). Omitted on screens with no file viewer, where
|
||||
* paths render as plain text (no behavior change). */
|
||||
onOpenFile?: (pathText: string) => void
|
||||
/** Native-chat assistant prose only: renders `::orca-visual{...}` directive lines. Without it,
|
||||
* a directive line is ordinary text. Must be referentially stable (this component is memoized). */
|
||||
renderVisual?: (directive: NativeChatVisualDirective, index: number) => ReactNode
|
||||
}
|
||||
|
||||
const MAX_TABLE_ROWS = 40
|
||||
@@ -233,14 +236,14 @@ function MobileMarkdownContent({
|
||||
fallback = '',
|
||||
rangeSelectable = false,
|
||||
textScale = 1,
|
||||
onOpenFile
|
||||
onOpenFile,
|
||||
renderVisual
|
||||
}: Props) {
|
||||
// Interactive children own their touches and must forward the row action.
|
||||
const setup = useContext(MarkdownTextContext)
|
||||
const rowLongPress = setup.androidTranscript ? setup.onLongPress : undefined
|
||||
const text = content?.trim() ?? ''
|
||||
const previewText = useMemo(() => normalizeMobileMarkdownPreviewHtml(text), [text])
|
||||
const blocks = useMemo(() => parseMobileMarkdown(previewText), [previewText])
|
||||
const { blocks, directives } = useMobileMarkdownBlocks(text, renderVisual !== undefined)
|
||||
// Scale prose sizes; inline spans inherit fontSize from the wrapping Text.
|
||||
const scaled = (size: number): { fontSize: number; lineHeight: number } | null =>
|
||||
textScale !== 1 ? { fontSize: size * textScale, lineHeight: (size + 6) * textScale } : null
|
||||
@@ -259,6 +262,14 @@ function MobileMarkdownContent({
|
||||
return (
|
||||
<View style={styles.root}>
|
||||
{blocks.map((block, index) => {
|
||||
if (block.type === 'visual') {
|
||||
const directive = directives[block.index]
|
||||
return directive && renderVisual ? (
|
||||
<Fragment key={`visual:${block.index}:${directive.file}`}>
|
||||
{renderVisual(directive, block.index)}
|
||||
</Fragment>
|
||||
) : null
|
||||
}
|
||||
if (block.type === 'heading') {
|
||||
return (
|
||||
<MarkdownText
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { memo } from 'react'
|
||||
import { StyleSheet, Text, View } from 'react-native'
|
||||
import type { RuntimeWorktreeAgentRow } from '../../../src/shared/runtime-types'
|
||||
import { withoutNativeChatVisualDirectiveLines } from '../../../src/shared/native-chat-visual-directive'
|
||||
import { colors, spacing } from '../theme/mobile-theme'
|
||||
import {
|
||||
agentDisplayLabel,
|
||||
@@ -26,7 +27,15 @@ type Props = {
|
||||
// Mirrors desktop DashboardAgentRow's compact in-card layout.
|
||||
function WorktreeAgentRowComponent({ agent, depth, now, unvisited }: Props) {
|
||||
const dotState = agentDotState(agent, now)
|
||||
const label = agentDisplayLabel(agent, now)
|
||||
// A visual line renders only in the transcript; the row shows the words around it, and a reply
|
||||
// that is only a visual falls back like an empty one.
|
||||
const reply = agent.lastAssistantMessage
|
||||
const label = agentDisplayLabel(
|
||||
reply
|
||||
? { ...agent, lastAssistantMessage: withoutNativeChatVisualDirectiveLines(reply) }
|
||||
: agent,
|
||||
now
|
||||
)
|
||||
const ts = formatTimeAgo(agentRowTimeAt(agent), now)
|
||||
|
||||
return (
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
import { createElement } from 'react'
|
||||
import { act, create, type ReactTestRenderer } from 'react-test-renderer'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { RuntimeWorktreeAgentRow } from '../../../src/shared/runtime-types'
|
||||
import { WorktreeAgentRow } from './WorktreeAgentRow'
|
||||
|
||||
vi.mock('react-native', () => ({
|
||||
StyleSheet: { create: <T,>(styles: T) => styles },
|
||||
Text: 'Text',
|
||||
View: 'View'
|
||||
}))
|
||||
vi.mock('./AgentStateDot', () => ({ AgentStateDot: () => null }))
|
||||
vi.mock('./MobileAgentIcon', () => ({ MobileAgentIcon: () => null }))
|
||||
|
||||
function agent(lastAssistantMessage: string | null): RuntimeWorktreeAgentRow {
|
||||
return {
|
||||
paneKey: 'tab:leaf',
|
||||
parentPaneKey: null,
|
||||
state: 'done',
|
||||
agentType: 'claude',
|
||||
prompt: 'Show usage by day',
|
||||
taskTitle: null,
|
||||
displayName: null,
|
||||
lastAssistantMessage,
|
||||
toolName: null,
|
||||
toolInput: null,
|
||||
interrupted: false,
|
||||
stateStartedAt: 1_000,
|
||||
updatedAt: 1_000
|
||||
}
|
||||
}
|
||||
|
||||
describe('WorktreeAgentRow visual lines', () => {
|
||||
let renderer: ReactTestRenderer | null = null
|
||||
|
||||
afterEach(() => {
|
||||
act(() => renderer?.unmount())
|
||||
renderer = null
|
||||
})
|
||||
|
||||
function label(row: RuntimeWorktreeAgentRow): string {
|
||||
act(() => {
|
||||
renderer = create(
|
||||
createElement(WorktreeAgentRow, { agent: row, depth: 0, now: 2_000, unvisited: false })
|
||||
)
|
||||
})
|
||||
return String(renderer!.root.findAll((node) => String(node.type) === 'Text')[0]!.props.children)
|
||||
}
|
||||
|
||||
it('shows the words around a visual line, never the line itself', () => {
|
||||
expect(label(agent('Here it is.\n::orca-visual{file="usage.html"}\nSunday dips.'))).toBe(
|
||||
'Here it is.\nSunday dips.'
|
||||
)
|
||||
})
|
||||
|
||||
it('falls back to the prompt when the reply is only a visual', () => {
|
||||
expect(label(agent('::orca-visual{file="usage.html" title="Usage"}'))).toBe('Show usage by day')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,17 @@
|
||||
// JSON.stringify escapes quotes and control chars but leaves `<`, `>`, `&`, and
|
||||
// the U+2028/U+2029 line separators raw — so a value containing `</script>` would
|
||||
// close the inline <script> this is spliced into and let the rest execute as
|
||||
// markup. These characters only ever appear inside JSON string literals, so
|
||||
// escaping them to \uXXXX is always valid and always parses back to the exact
|
||||
// original text inside the WebView.
|
||||
export function encodeJsonForScript(json: string): string {
|
||||
return json.replace(
|
||||
/[<>&\u2028\u2029]/g,
|
||||
(char) => `\\u${char.charCodeAt(0).toString(16).padStart(4, '0')}`
|
||||
)
|
||||
}
|
||||
|
||||
/** A value as a JavaScript literal safe to splice into an inline <script>. */
|
||||
export function inlineScriptLiteral(value: unknown): string {
|
||||
return encodeJsonForScript(JSON.stringify(value))
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
import { isTableSeparator, splitTableRow } from './rich-markdown/markdown-table-rows'
|
||||
import { mobileMarkdownVisualPlaceholderIndex } from './mobile-markdown-visual-lines'
|
||||
|
||||
export type MobileMarkdownBlock =
|
||||
| { type: 'paragraph'; text: string }
|
||||
@@ -9,11 +10,17 @@ export type MobileMarkdownBlock =
|
||||
| { type: 'image'; alt: string; url: string }
|
||||
| { type: 'table'; headers: string[]; rows: string[][] }
|
||||
| { type: 'rule' }
|
||||
/** A native-chat visual directive; `index` is its position in the protected directive list. */
|
||||
| { type: 'visual'; index: number }
|
||||
|
||||
const HEADING = /^(#{1,6})\s+(.+)$/
|
||||
const CODE_FENCE = /^```([A-Za-z0-9_-]+)?\s*$/
|
||||
|
||||
export function parseMobileMarkdown(content: string): MobileMarkdownBlock[] {
|
||||
/** `visualCount`: how many directive placeholders `protectMobileMarkdownVisualLines` left in
|
||||
* `content`; zero (the default) leaves every line to the ordinary rules. */
|
||||
export function parseMobileMarkdown(content: string, visualCount = 0): MobileMarkdownBlock[] {
|
||||
const visualIndex = (line: string): number | null =>
|
||||
visualCount > 0 ? mobileMarkdownVisualPlaceholderIndex(line, visualCount) : null
|
||||
const lines = content.replace(/\r\n?/g, '\n').split('\n')
|
||||
const blocks: MobileMarkdownBlock[] = []
|
||||
let index = 0
|
||||
@@ -42,6 +49,13 @@ export function parseMobileMarkdown(content: string): MobileMarkdownBlock[] {
|
||||
continue
|
||||
}
|
||||
|
||||
const visual = visualIndex(line)
|
||||
if (visual !== null) {
|
||||
blocks.push({ type: 'visual', index: visual })
|
||||
index += 1
|
||||
continue
|
||||
}
|
||||
|
||||
if (/^\s*(-{3,}|\*{3,}|_{3,})\s*$/.test(line)) {
|
||||
blocks.push({ type: 'rule' })
|
||||
index += 1
|
||||
@@ -116,7 +130,8 @@ export function parseMobileMarkdown(content: string): MobileMarkdownBlock[] {
|
||||
!HEADING.test(lines[index] ?? '') &&
|
||||
!/^>\s?/.test(lines[index] ?? '') &&
|
||||
!/^\s*(?:[-*+]|\d+[.)])\s+/.test(lines[index] ?? '') &&
|
||||
!/^\s*(-{3,}|\*{3,}|_{3,})\s*$/.test(lines[index] ?? '')
|
||||
!/^\s*(-{3,}|\*{3,}|_{3,})\s*$/.test(lines[index] ?? '') &&
|
||||
visualIndex(lines[index] ?? '') === null
|
||||
) {
|
||||
paragraph.push(lines[index] ?? '')
|
||||
index += 1
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { NATIVE_CHAT_VISUAL_MAX_PER_MESSAGE } from '../../../src/shared/native-chat-visual-directive'
|
||||
import { normalizeMobileMarkdownPreviewHtml } from './mobile-markdown-preview-html'
|
||||
import { parseMobileMarkdown } from './mobile-markdown-parser'
|
||||
import { protectMobileMarkdownVisualLines } from './mobile-markdown-visual-lines'
|
||||
|
||||
/** The same steps MobileMarkdown runs when a transcript renders visuals. */
|
||||
function blocksOf(content: string) {
|
||||
const { text, directives } = protectMobileMarkdownVisualLines(content)
|
||||
return {
|
||||
directives,
|
||||
blocks: parseMobileMarkdown(normalizeMobileMarkdownPreviewHtml(text), directives.length)
|
||||
}
|
||||
}
|
||||
|
||||
describe('native-chat visual lines in mobile markdown', () => {
|
||||
it('turns a directive line into its own block between prose', () => {
|
||||
const { directives, blocks } = blocksOf(
|
||||
'Here is the chart.\n::orca-visual{file="usage-3f2a.html" title="Usage"}\nIt shows a dip.'
|
||||
)
|
||||
expect(directives).toEqual([{ file: 'usage-3f2a.html', title: 'Usage' }])
|
||||
expect(blocks).toEqual([
|
||||
{ type: 'paragraph', text: 'Here is the chart.' },
|
||||
{ type: 'visual', index: 0 },
|
||||
{ type: 'paragraph', text: 'It shows a dip.' }
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps the title exactly as written, before entity decoding and tag stripping', () => {
|
||||
const { directives } = blocksOf('::orca-visual{file="a.html" title="<b>Q1</b> & Q2"}')
|
||||
expect(directives).toEqual([{ file: 'a.html', title: '<b>Q1</b> & Q2' }])
|
||||
})
|
||||
|
||||
it('accepts CRLF line endings and up to three leading spaces', () => {
|
||||
const { blocks } = blocksOf('intro\r\n ::orca-visual{file="a.html"}\r\nend')
|
||||
expect(blocks.map((block) => block.type)).toEqual(['paragraph', 'visual', 'paragraph'])
|
||||
})
|
||||
|
||||
it('leaves directives inside fenced code, quotes, list items and inline code as text', () => {
|
||||
const fenced = blocksOf('```md\n::orca-visual{file="a.html"}\n```')
|
||||
expect(fenced.directives).toEqual([])
|
||||
expect(fenced.blocks).toEqual([
|
||||
{ type: 'code', text: '::orca-visual{file="a.html"}', language: 'md', closed: true }
|
||||
])
|
||||
expect(blocksOf('> ::orca-visual{file="a.html"}').directives).toEqual([])
|
||||
expect(blocksOf('- ::orca-visual{file="a.html"}').directives).toEqual([])
|
||||
expect(blocksOf('see `::orca-visual{file="a.html"}`').directives).toEqual([])
|
||||
expect(blocksOf(' ::orca-visual{file="a.html"}').directives).toEqual([])
|
||||
})
|
||||
|
||||
it('resumes recognizing directives after a fence closes', () => {
|
||||
const { directives } = blocksOf('```\ncode\n```\n::orca-visual{file="after.html"}')
|
||||
expect(directives).toEqual([{ file: 'after.html', title: null }])
|
||||
})
|
||||
|
||||
it('leaves malformed or path-like directives as literal text', () => {
|
||||
for (const line of [
|
||||
'::orca-visual{file="../secret.html"}',
|
||||
'::orca-visual{file="a.html" file="b.html"}',
|
||||
'::orca-visual{file="a.htm"}',
|
||||
'::orca-visual{file="a.html"',
|
||||
'::orca-visual{file="a.html"} trailing words'
|
||||
]) {
|
||||
const { directives, blocks } = blocksOf(line)
|
||||
expect(directives, line).toEqual([])
|
||||
expect(blocks, line).toEqual([{ type: 'paragraph', text: line }])
|
||||
}
|
||||
})
|
||||
|
||||
it('renders directives past the per-message cap as text', () => {
|
||||
const lines = Array.from(
|
||||
{ length: NATIVE_CHAT_VISUAL_MAX_PER_MESSAGE + 1 },
|
||||
(_, index) => `::orca-visual{file="v${index}.html"}`
|
||||
)
|
||||
const { directives, blocks } = blocksOf(lines.join('\n'))
|
||||
expect(directives).toHaveLength(NATIVE_CHAT_VISUAL_MAX_PER_MESSAGE)
|
||||
expect(blocks.at(-1)).toEqual({ type: 'paragraph', text: lines.at(-1) })
|
||||
})
|
||||
|
||||
it('renders no visuals for text that already spells a placeholder', () => {
|
||||
const { directives } = blocksOf('\uE000ORCA_VISUAL_0\uE000\n::orca-visual{file="a.html"}')
|
||||
expect(directives).toEqual([])
|
||||
})
|
||||
|
||||
it('never treats a placeholder-looking line as a visual when none were protected', () => {
|
||||
expect(parseMobileMarkdown('\uE000ORCA_VISUAL_0\uE000')).toEqual([
|
||||
{ type: 'paragraph', text: '\uE000ORCA_VISUAL_0\uE000' }
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,69 @@
|
||||
import {
|
||||
NATIVE_CHAT_VISUAL_MAX_PER_MESSAGE,
|
||||
parseNativeChatVisualDirectiveLine,
|
||||
type NativeChatVisualDirective
|
||||
} from '../../../src/shared/native-chat-visual-directive'
|
||||
|
||||
/** Same fence shapes the block parser opens and closes on, so a directive inside code stays code. */
|
||||
const CODE_FENCE_OPEN = /^```([A-Za-z0-9_-]+)?\s*$/
|
||||
const CODE_FENCE_CLOSE = /^```\s*$/
|
||||
// Private-use delimiters, as the preview normalizer's own code placeholders use.
|
||||
const PLACEHOLDER_PREFIX = '\uE000ORCA_VISUAL_'
|
||||
const PLACEHOLDER = /^\uE000ORCA_VISUAL_(\d+)\uE000$/
|
||||
|
||||
export type MobileMarkdownVisualLines = {
|
||||
/** The content with each recognized directive line replaced by a placeholder line. */
|
||||
text: string
|
||||
directives: NativeChatVisualDirective[]
|
||||
}
|
||||
|
||||
function placeholderFor(index: number): string {
|
||||
return `${PLACEHOLDER_PREFIX}${index}\uE000`
|
||||
}
|
||||
|
||||
/**
|
||||
* Lifts native-chat visual directive lines out of assistant prose before preview normalization,
|
||||
* which decodes entities and strips tags and would otherwise rewrite a title. Only top-level lines
|
||||
* outside fenced code count; quotes and list items never start with the marker. Past the per-message
|
||||
* cap a directive stays literal text.
|
||||
*/
|
||||
export function protectMobileMarkdownVisualLines(content: string): MobileMarkdownVisualLines {
|
||||
// Text that already spells a placeholder would alias a real one; it renders no visuals at all.
|
||||
if (content.includes(PLACEHOLDER_PREFIX)) {
|
||||
return { text: content, directives: [] }
|
||||
}
|
||||
const lines = content.replace(/\r\n?/g, '\n').split('\n')
|
||||
const directives: NativeChatVisualDirective[] = []
|
||||
let inFence = false
|
||||
const next = lines.map((line) => {
|
||||
if (inFence) {
|
||||
inFence = !CODE_FENCE_CLOSE.test(line)
|
||||
return line
|
||||
}
|
||||
if (CODE_FENCE_OPEN.test(line)) {
|
||||
inFence = true
|
||||
return line
|
||||
}
|
||||
if (directives.length >= NATIVE_CHAT_VISUAL_MAX_PER_MESSAGE) {
|
||||
return line
|
||||
}
|
||||
const directive = parseNativeChatVisualDirectiveLine(line)
|
||||
if (!directive) {
|
||||
return line
|
||||
}
|
||||
directives.push(directive)
|
||||
// Blank lines around it make the directive its own block, splitting any paragraph it was in.
|
||||
return `\n${placeholderFor(directives.length - 1)}\n`
|
||||
})
|
||||
return { text: next.join('\n'), directives }
|
||||
}
|
||||
|
||||
/** The directive index a parsed line stands for, or null for any other line. */
|
||||
export function mobileMarkdownVisualPlaceholderIndex(line: string, count: number): number | null {
|
||||
const match = PLACEHOLDER.exec(line.trim())
|
||||
if (!match) {
|
||||
return null
|
||||
}
|
||||
const index = Number(match[1])
|
||||
return index < count ? index : null
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import { WebView } from 'react-native-webview'
|
||||
import { colors, radii, spacing, typography } from '../../theme/mobile-theme'
|
||||
import { MERMAID_DIAGRAM_CONFIG } from './mermaid-diagram-config'
|
||||
import { MERMAID_ENGINE_JS } from './mermaid-webview-engine.generated'
|
||||
import { encodeJsonForScript } from '../inline-script-json'
|
||||
|
||||
export type MermaidDiagramProps = {
|
||||
source: string
|
||||
@@ -79,19 +80,6 @@ function MermaidFallback({ source, base }: MermaidDiagramProps) {
|
||||
)
|
||||
}
|
||||
|
||||
// JSON.stringify escapes quotes and control chars but leaves `<`, `>`, `&`, and
|
||||
// the U+2028/U+2029 line separators raw — so a value containing `</script>` would
|
||||
// close the inline <script> this is spliced into and let the rest execute as
|
||||
// markup. These characters only ever appear inside JSON string literals, so
|
||||
// escaping them to \uXXXX is always valid and always parses back to the exact
|
||||
// original text inside the WebView.
|
||||
function encodeJsonForScript(json: string): string {
|
||||
return json.replace(
|
||||
/[<>&\u2028\u2029]/g,
|
||||
(char) => `\\u${char.charCodeAt(0).toString(16).padStart(4, '0')}`
|
||||
)
|
||||
}
|
||||
|
||||
// Diagram source is untrusted: agent output, PR and chat content.
|
||||
function encodeSourceForScript(source: string): string {
|
||||
return encodeJsonForScript(JSON.stringify(source))
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
import { useMemo } from 'react'
|
||||
import type { NativeChatVisualDirective } from '../../../src/shared/native-chat-visual-directive'
|
||||
import { normalizeMobileMarkdownPreviewHtml } from './mobile-markdown-preview-html'
|
||||
import { parseMobileMarkdown, type MobileMarkdownBlock } from './mobile-markdown-parser'
|
||||
import { protectMobileMarkdownVisualLines } from './mobile-markdown-visual-lines'
|
||||
|
||||
const NO_DIRECTIVES: NativeChatVisualDirective[] = []
|
||||
|
||||
/** The blocks MobileMarkdown draws; with `visuals`, directive lines become `visual` blocks. */
|
||||
export function useMobileMarkdownBlocks(
|
||||
text: string,
|
||||
visuals: boolean
|
||||
): { blocks: MobileMarkdownBlock[]; directives: NativeChatVisualDirective[] } {
|
||||
const visualLines = useMemo(
|
||||
() => (visuals ? protectMobileMarkdownVisualLines(text) : null),
|
||||
[text, visuals]
|
||||
)
|
||||
const previewText = useMemo(
|
||||
() => normalizeMobileMarkdownPreviewHtml(visualLines?.text ?? text),
|
||||
[visualLines, text]
|
||||
)
|
||||
const directives = visualLines?.directives ?? NO_DIRECTIVES
|
||||
const blocks = useMemo(
|
||||
() => parseMobileMarkdown(previewText, directives.length),
|
||||
[previewText, directives.length]
|
||||
)
|
||||
return { blocks, directives }
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
import { MobileSelectableText as Text } from '../components/MobileSelectableText'
|
||||
import { memo, useCallback, useState } from 'react'
|
||||
import { memo, useCallback, useContext, useState } from 'react'
|
||||
import { Image, Text as NativeText, View } from 'react-native'
|
||||
import { INLINE_TEXT_SELECTION } from '../components/inline-text-selection'
|
||||
import { MobileNativeChatMessageActionsSheet } from './MobileNativeChatMessageActionsSheet'
|
||||
@@ -21,17 +21,28 @@ import type { NativeChatTurnStatus } from './use-mobile-native-chat-turn-status'
|
||||
import { isRenderableImageUri } from './mobile-native-chat-image-preview'
|
||||
import { styles, TEXT_SIZE } from './mobile-native-chat-message-styles'
|
||||
import { agentMessageAttribution } from './mobile-agent-message-attribution'
|
||||
import { withoutPendingNativeChatVisualDirectiveTail } from '../../../src/shared/native-chat-visual-directive'
|
||||
import {
|
||||
MobileNativeChatVisualContext,
|
||||
type MobileNativeChatVisualRender
|
||||
} from './mobile-native-chat-visual-context'
|
||||
|
||||
function Prose({
|
||||
block,
|
||||
invert,
|
||||
fontScale,
|
||||
onOpenFile,
|
||||
onLongPress
|
||||
onLongPress,
|
||||
renderVisual,
|
||||
holdPendingVisual = false
|
||||
}: {
|
||||
block: NativeChatBlock
|
||||
invert?: boolean
|
||||
fontScale: number
|
||||
/** Assistant prose of a structured chat only. */
|
||||
renderVisual?: MobileNativeChatVisualRender
|
||||
/** The reply may still be growing: a directive still being typed at its tail is held back. */
|
||||
holdPendingVisual?: boolean
|
||||
onOpenFile?: (relativePath: string) => void
|
||||
/** Android only: routes a long press on a link span to the row's actions sheet. */
|
||||
onLongPress?: () => void
|
||||
@@ -61,7 +72,12 @@ function Prose({
|
||||
}
|
||||
return (
|
||||
<MobileMarkdown
|
||||
content={block.text}
|
||||
content={
|
||||
holdPendingVisual && renderVisual
|
||||
? withoutPendingNativeChatVisualDirectiveTail(block.text)
|
||||
: block.text
|
||||
}
|
||||
renderVisual={renderVisual}
|
||||
rangeSelectable
|
||||
textScale={1.25 * fontScale}
|
||||
onOpenFile={onOpenFile}
|
||||
@@ -94,6 +110,7 @@ function Prose({
|
||||
|
||||
function MobileNativeChatMessageImpl({
|
||||
message,
|
||||
mayStillGrow = false,
|
||||
toolsExpanded = false,
|
||||
fontScale = 1,
|
||||
onOpenFile,
|
||||
@@ -109,6 +126,8 @@ function MobileNativeChatMessageImpl({
|
||||
onToggleReasoning
|
||||
}: {
|
||||
message: NativeChatMessage
|
||||
/** The newest assistant row of a live turn with no prompt open: its last text may still grow. */
|
||||
mayStillGrow?: boolean
|
||||
toolsExpanded?: boolean
|
||||
/** Multiplies all chat text sizes for pinch-to-zoom (1 = no change). */
|
||||
fontScale?: number
|
||||
@@ -160,6 +179,12 @@ function MobileNativeChatMessageImpl({
|
||||
// Keep the memoized Markdown context stable as the message streams.
|
||||
const openActions = useCallback(() => setActionsOpen(true), [])
|
||||
const onLongPress = INLINE_TEXT_SELECTION ? undefined : openActions
|
||||
const renderVisual = useContext(MobileNativeChatVisualContext) ?? undefined
|
||||
// Structured replies grow in place: only the last block of the newest row may still be typing.
|
||||
const growingBlock =
|
||||
message.role === 'assistant' && mayStillGrow && activeTurnIsWorking === true
|
||||
? message.blocks.at(-1)
|
||||
: undefined
|
||||
|
||||
const statusRow = turnStatus ? (
|
||||
<MobileNativeChatTurnStatus
|
||||
@@ -228,6 +253,8 @@ function MobileNativeChatMessageImpl({
|
||||
fontScale={fontScale}
|
||||
onOpenFile={onOpenFile}
|
||||
onLongPress={onLongPress}
|
||||
renderVisual={message.role === 'assistant' ? renderVisual : undefined}
|
||||
holdPendingVisual={block === growingBlock}
|
||||
/>
|
||||
))}
|
||||
{showToolRun ? (
|
||||
|
||||
@@ -12,6 +12,7 @@ vi.mock('react-native', () => ({
|
||||
|
||||
vi.mock('./MobileNativeChatView', () => ({ MobileNativeChatView: 'ChatView' }))
|
||||
vi.mock('./MobileNativeChatQueuedMessages', () => ({ MobileNativeChatQueuedMessages: 'Queued' }))
|
||||
vi.mock('./MobileNativeChatVisual', () => ({ useMobileNativeChatVisualRenderer: () => null }))
|
||||
|
||||
function assistantTurn(id: string, text: string): NativeChatMessage {
|
||||
return { id, role: 'assistant', blocks: [{ type: 'text', text }], timestamp: 0, source: 'hook' }
|
||||
|
||||
@@ -6,6 +6,8 @@ import type { MobileNativeChatImageAttachments } from './use-mobile-native-chat-
|
||||
import type { MobileNativeChatController } from './use-mobile-native-chat-controller'
|
||||
import { useMobileNativeChatStreamingBubble } from './use-mobile-native-chat-streaming-bubble'
|
||||
import { useMobileNativeChatQueuedSlot } from './use-mobile-native-chat-queued-slot'
|
||||
import { useMobileNativeChatVisualRenderer } from './MobileNativeChatVisual'
|
||||
import { MobileNativeChatVisualContext } from './mobile-native-chat-visual-context'
|
||||
|
||||
type Props = {
|
||||
controller: MobileNativeChatController
|
||||
@@ -73,71 +75,74 @@ export function MobileNativeChatOverlay({
|
||||
// The indicator's `stopping` is the display status, decided once in the session hook.
|
||||
steerHeld: controller.nativeChatTurnIndicator?.stopping === true
|
||||
})
|
||||
const visuals = useMobileNativeChatVisualRenderer(controller.nativeChatVisualSource)
|
||||
if (!controller.showNativeChat) {
|
||||
return null
|
||||
}
|
||||
return (
|
||||
<View style={styles.overlay}>
|
||||
<MobileNativeChatView
|
||||
messages={session.messages}
|
||||
folded={folded}
|
||||
status={session.status}
|
||||
error={session.error}
|
||||
readFailedFinally={session.readFailedFinally === true}
|
||||
agent={controller.nativeChatAgent}
|
||||
agentWorking={controller.nativeChatAgentWorking}
|
||||
canStop={controller.nativeChatCanStop}
|
||||
structuredActivityUi={controller.nativeChatStructured}
|
||||
turnIndicator={controller.nativeChatTurnIndicator}
|
||||
workingStartedAt={controller.nativeChatWorkingStartedAt}
|
||||
settledTurns={controller.nativeChatSettledTurns}
|
||||
turnJournal={controller.nativeChatTurnJournal}
|
||||
streaming={streaming}
|
||||
onStop={controller.handleNativeChatStop}
|
||||
ask={controller.nativeChatAsk}
|
||||
askKey={controller.nativeChatAskKey}
|
||||
promptKey={controller.nativeChatPromptKey}
|
||||
onDismissAsk={controller.dismissNativeChatAsk}
|
||||
onAnswerAsk={controller.handleNativeChatAnswerAsk}
|
||||
onCancelAsk={controller.handleNativeChatCancelAsk}
|
||||
onCancelPrompt={controller.handleNativeChatCancelPrompt}
|
||||
onCollapseAsk={controller.collapseNativeChatAsk}
|
||||
onCollapsePrompt={controller.collapseNativeChatPrompt}
|
||||
collapsedPrompt={controller.nativeChatCollapsedPrompt}
|
||||
question={controller.nativeChatQuestion}
|
||||
onAnswerQuestion={controller.handleNativeChatQuestionAnswer}
|
||||
permission={controller.nativeChatPermission}
|
||||
onRespondPermission={controller.handleNativeChatRespondPermission}
|
||||
queuedSlot={queuedSlot}
|
||||
onOpenFile={onOpenFile}
|
||||
hasMore={session.hasMore}
|
||||
loadingEarlier={session.loadingEarlier}
|
||||
onLoadEarlier={session.loadEarlier}
|
||||
onSend={images.sendNativeChat}
|
||||
sendSurfaceId={sendSurfaceId}
|
||||
getSendCompletionGeneration={getSendCompletionGeneration}
|
||||
getComposerEditGeneration={controller.getChatComposerEditGeneration}
|
||||
pending={controller.chatPending}
|
||||
imagePreviewsByMessageId={controller.chatImagePreviewsByMessageId}
|
||||
composerText={controller.chatComposerText}
|
||||
onComposerTextChange={controller.setChatComposerText}
|
||||
onAttachImage={() => void images.attachImage('library')}
|
||||
attachments={images.attachments}
|
||||
onRemoveAttachment={images.removeAttachment}
|
||||
isAttaching={images.isAttaching}
|
||||
onMicPress={onMicPress}
|
||||
micActive={micActive}
|
||||
dictationMode={dictationMode}
|
||||
onMicPressIn={onMicPressIn}
|
||||
onMicPressOut={onMicPressOut}
|
||||
inputLockReason={inputLockReason}
|
||||
sendErrorMessage={sendErrorMessage}
|
||||
onClearSendError={onClearSendError}
|
||||
filePaths={controller.nativeChatFilePaths}
|
||||
onNeedFiles={controller.loadNativeChatFiles}
|
||||
sessionOptions={controller.nativeChatSessionOptions}
|
||||
keyboardInset={keyboardInset}
|
||||
/>
|
||||
<MobileNativeChatVisualContext.Provider value={visuals}>
|
||||
<MobileNativeChatView
|
||||
messages={session.messages}
|
||||
folded={folded}
|
||||
status={session.status}
|
||||
error={session.error}
|
||||
readFailedFinally={session.readFailedFinally === true}
|
||||
agent={controller.nativeChatAgent}
|
||||
agentWorking={controller.nativeChatAgentWorking}
|
||||
canStop={controller.nativeChatCanStop}
|
||||
structuredActivityUi={controller.nativeChatStructured}
|
||||
turnIndicator={controller.nativeChatTurnIndicator}
|
||||
workingStartedAt={controller.nativeChatWorkingStartedAt}
|
||||
settledTurns={controller.nativeChatSettledTurns}
|
||||
turnJournal={controller.nativeChatTurnJournal}
|
||||
streaming={streaming}
|
||||
onStop={controller.handleNativeChatStop}
|
||||
ask={controller.nativeChatAsk}
|
||||
askKey={controller.nativeChatAskKey}
|
||||
promptKey={controller.nativeChatPromptKey}
|
||||
onDismissAsk={controller.dismissNativeChatAsk}
|
||||
onAnswerAsk={controller.handleNativeChatAnswerAsk}
|
||||
onCancelAsk={controller.handleNativeChatCancelAsk}
|
||||
onCancelPrompt={controller.handleNativeChatCancelPrompt}
|
||||
onCollapseAsk={controller.collapseNativeChatAsk}
|
||||
onCollapsePrompt={controller.collapseNativeChatPrompt}
|
||||
collapsedPrompt={controller.nativeChatCollapsedPrompt}
|
||||
question={controller.nativeChatQuestion}
|
||||
onAnswerQuestion={controller.handleNativeChatQuestionAnswer}
|
||||
permission={controller.nativeChatPermission}
|
||||
onRespondPermission={controller.handleNativeChatRespondPermission}
|
||||
queuedSlot={queuedSlot}
|
||||
onOpenFile={onOpenFile}
|
||||
hasMore={session.hasMore}
|
||||
loadingEarlier={session.loadingEarlier}
|
||||
onLoadEarlier={session.loadEarlier}
|
||||
onSend={images.sendNativeChat}
|
||||
sendSurfaceId={sendSurfaceId}
|
||||
getSendCompletionGeneration={getSendCompletionGeneration}
|
||||
getComposerEditGeneration={controller.getChatComposerEditGeneration}
|
||||
pending={controller.chatPending}
|
||||
imagePreviewsByMessageId={controller.chatImagePreviewsByMessageId}
|
||||
composerText={controller.chatComposerText}
|
||||
onComposerTextChange={controller.setChatComposerText}
|
||||
onAttachImage={() => void images.attachImage('library')}
|
||||
attachments={images.attachments}
|
||||
onRemoveAttachment={images.removeAttachment}
|
||||
isAttaching={images.isAttaching}
|
||||
onMicPress={onMicPress}
|
||||
micActive={micActive}
|
||||
dictationMode={dictationMode}
|
||||
onMicPressIn={onMicPressIn}
|
||||
onMicPressOut={onMicPressOut}
|
||||
inputLockReason={inputLockReason}
|
||||
sendErrorMessage={sendErrorMessage}
|
||||
onClearSendError={onClearSendError}
|
||||
filePaths={controller.nativeChatFilePaths}
|
||||
onNeedFiles={controller.loadNativeChatFiles}
|
||||
sessionOptions={controller.nativeChatSessionOptions}
|
||||
keyboardInset={keyboardInset}
|
||||
/>
|
||||
</MobileNativeChatVisualContext.Provider>
|
||||
</View>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
import { memo, useCallback, useMemo, useState } from 'react'
|
||||
import { ActivityIndicator, Modal, Pressable, StyleSheet, Text, View } from 'react-native'
|
||||
import { useSafeAreaInsets } from 'react-native-safe-area-context'
|
||||
import { Maximize2, X } from 'lucide-react-native'
|
||||
import type { NativeChatVisualDirective } from '../../../src/shared/native-chat-visual-directive'
|
||||
import { colors, radii, spacing, typography } from '../theme/mobile-theme'
|
||||
import type { MobileNativeChatVisualSource } from './mobile-native-chat-visual-read'
|
||||
import { MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT } from './mobile-native-chat-visual-host-document'
|
||||
import { MobileNativeChatVisualFrame } from './MobileNativeChatVisualFrame'
|
||||
import { useMobileNativeChatVisual } from './use-mobile-native-chat-visual'
|
||||
import type { MobileNativeChatVisualRender } from './mobile-native-chat-visual-context'
|
||||
|
||||
const DEFAULT_TITLE = 'Visualization'
|
||||
|
||||
/** The quiet reserved space a visual takes while it loads. */
|
||||
function MobileNativeChatVisualPlaceholder() {
|
||||
return (
|
||||
<View style={[styles.frame, styles.placeholder]}>
|
||||
<ActivityIndicator size="small" color={colors.textMuted} />
|
||||
</View>
|
||||
)
|
||||
}
|
||||
|
||||
/** The transcript's visual renderer for a structured chat's source; null without one. */
|
||||
export function useMobileNativeChatVisualRenderer(
|
||||
source: MobileNativeChatVisualSource | null
|
||||
): MobileNativeChatVisualRender | null {
|
||||
return useMemo(
|
||||
() =>
|
||||
source
|
||||
? (directive) => <MobileNativeChatVisual directive={directive} source={source} />
|
||||
: null,
|
||||
[source]
|
||||
)
|
||||
}
|
||||
|
||||
/** A `::orca-visual{...}` line in a finished assistant reply, rendered from the chat's host. */
|
||||
export const MobileNativeChatVisual = memo(function MobileNativeChatVisual({
|
||||
directive,
|
||||
source
|
||||
}: {
|
||||
directive: NativeChatVisualDirective
|
||||
source: MobileNativeChatVisualSource
|
||||
}) {
|
||||
const { state, retry } = useMobileNativeChatVisual(source, directive.file)
|
||||
// A frame that navigated away or kept crashing; cleared by the reader's retry.
|
||||
const [frameFailed, setFrameFailed] = useState(false)
|
||||
const [fullscreen, setFullscreen] = useState(false)
|
||||
const title = directive.title ?? DEFAULT_TITLE
|
||||
const onFrameFailed = useCallback(() => {
|
||||
setFrameFailed(true)
|
||||
setFullscreen(false)
|
||||
}, [])
|
||||
const onRetry = useCallback(() => {
|
||||
setFrameFailed(false)
|
||||
retry()
|
||||
}, [retry])
|
||||
const closeFullscreen = useCallback(() => setFullscreen(false), [])
|
||||
|
||||
if (state.kind === 'loading') {
|
||||
return <MobileNativeChatVisualPlaceholder />
|
||||
}
|
||||
if (state.kind === 'unavailable' || frameFailed) {
|
||||
return (
|
||||
<Pressable
|
||||
onPress={onRetry}
|
||||
accessibilityRole="button"
|
||||
accessibilityLabel="Visualization unavailable"
|
||||
accessibilityHint="Tries to load it again"
|
||||
style={styles.unavailable}
|
||||
>
|
||||
<Text style={styles.unavailableText}>Visualization unavailable</Text>
|
||||
</Pressable>
|
||||
)
|
||||
}
|
||||
return (
|
||||
<View style={styles.frame}>
|
||||
<MobileNativeChatVisualFrame
|
||||
html={state.html}
|
||||
title={title}
|
||||
mode="inline"
|
||||
onFailed={onFrameFailed}
|
||||
/>
|
||||
<Pressable
|
||||
onPress={() => setFullscreen(true)}
|
||||
accessibilityRole="button"
|
||||
accessibilityLabel={`Open ${title} full screen`}
|
||||
hitSlop={8}
|
||||
style={styles.openButton}
|
||||
>
|
||||
<Maximize2 size={14} color={colors.textSecondary} strokeWidth={2.2} />
|
||||
</Pressable>
|
||||
{fullscreen ? (
|
||||
<MobileNativeChatVisualFullscreen
|
||||
html={state.html}
|
||||
title={title}
|
||||
onClose={closeFullscreen}
|
||||
onFailed={onFrameFailed}
|
||||
/>
|
||||
) : null}
|
||||
</View>
|
||||
)
|
||||
})
|
||||
|
||||
function MobileNativeChatVisualFullscreen({
|
||||
html,
|
||||
title,
|
||||
onClose,
|
||||
onFailed
|
||||
}: {
|
||||
html: string
|
||||
title: string
|
||||
onClose: () => void
|
||||
onFailed: () => void
|
||||
}) {
|
||||
const insets = useSafeAreaInsets()
|
||||
return (
|
||||
<Modal visible animationType="slide" presentationStyle="fullScreen" onRequestClose={onClose}>
|
||||
<View style={[styles.sheet, { paddingTop: insets.top, paddingBottom: insets.bottom }]}>
|
||||
<View style={styles.sheetHeader}>
|
||||
<Text style={styles.sheetTitle} numberOfLines={1}>
|
||||
{title}
|
||||
</Text>
|
||||
<Pressable
|
||||
onPress={onClose}
|
||||
accessibilityRole="button"
|
||||
accessibilityLabel="Close"
|
||||
hitSlop={8}
|
||||
style={styles.closeButton}
|
||||
>
|
||||
<X size={18} color={colors.textSecondary} strokeWidth={2.2} />
|
||||
</Pressable>
|
||||
</View>
|
||||
<MobileNativeChatVisualFrame
|
||||
html={html}
|
||||
title={title}
|
||||
mode="fullscreen"
|
||||
onFailed={onFailed}
|
||||
/>
|
||||
</View>
|
||||
</Modal>
|
||||
)
|
||||
}
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
frame: {
|
||||
borderWidth: StyleSheet.hairlineWidth,
|
||||
borderColor: colors.borderSubtle,
|
||||
borderRadius: radii.row,
|
||||
marginBottom: spacing.sm,
|
||||
overflow: 'hidden'
|
||||
},
|
||||
placeholder: {
|
||||
height: MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT,
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center'
|
||||
},
|
||||
unavailable: { marginBottom: spacing.sm, paddingVertical: spacing.xs },
|
||||
unavailableText: { color: colors.textMuted, fontSize: typography.metaSize },
|
||||
openButton: {
|
||||
position: 'absolute',
|
||||
top: spacing.xs,
|
||||
right: spacing.xs,
|
||||
width: 28,
|
||||
height: 28,
|
||||
borderRadius: 14,
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
backgroundColor: colors.bgPanel,
|
||||
borderWidth: StyleSheet.hairlineWidth,
|
||||
borderColor: colors.borderSubtle
|
||||
},
|
||||
sheet: { flex: 1, backgroundColor: colors.bgBase },
|
||||
sheetHeader: {
|
||||
flexDirection: 'row',
|
||||
alignItems: 'center',
|
||||
paddingHorizontal: spacing.md,
|
||||
paddingVertical: spacing.sm,
|
||||
borderBottomWidth: StyleSheet.hairlineWidth,
|
||||
borderBottomColor: colors.borderSubtle
|
||||
},
|
||||
sheetTitle: {
|
||||
flex: 1,
|
||||
color: colors.textPrimary,
|
||||
fontSize: typography.bodySize,
|
||||
fontWeight: '600'
|
||||
},
|
||||
closeButton: { padding: spacing.xs }
|
||||
})
|
||||
@@ -0,0 +1,200 @@
|
||||
import { memo, useCallback, useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { StyleSheet } from 'react-native'
|
||||
import { WebView, type WebViewMessageEvent } from 'react-native-webview'
|
||||
import type { ShouldStartLoadRequest } from 'react-native-webview/lib/WebViewTypes'
|
||||
import * as ExpoCrypto from 'expo-crypto'
|
||||
import { buildNativeChatVisualDocument } from '../../../src/shared/native-chat-visual-shell'
|
||||
import { createNativeChatVisualHeightGovernor } from '../../../src/shared/native-chat-visual-height-governor'
|
||||
import { openExternalLink } from '../platform/external-link'
|
||||
import {
|
||||
MOBILE_NATIVE_CHAT_VISUAL_LINK_INTERVAL_MS,
|
||||
readMobileNativeChatVisualBridgeMessage
|
||||
} from './mobile-native-chat-visual-bridge'
|
||||
import {
|
||||
buildMobileNativeChatVisualHostDocument,
|
||||
MOBILE_NATIVE_CHAT_VISUAL_APPLY_HEIGHT,
|
||||
MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT,
|
||||
type MobileNativeChatVisualHostMode
|
||||
} from './mobile-native-chat-visual-host-document'
|
||||
import { MOBILE_NATIVE_CHAT_VISUAL_THEME } from './mobile-native-chat-visual-theme'
|
||||
|
||||
export type MobileNativeChatVisualFrameProps = {
|
||||
html: string
|
||||
title: string
|
||||
mode: MobileNativeChatVisualHostMode
|
||||
/** The frame cannot show this visual (it navigated away, or its web process keeps dying). */
|
||||
onFailed: () => void
|
||||
}
|
||||
|
||||
/** One automatic reload after the web process dies; a page that kills it again is unavailable. */
|
||||
const MAX_PROCESS_RESTARTS = 1
|
||||
|
||||
function randomHex(): string {
|
||||
return Array.from(ExpoCrypto.getRandomBytes(16), (byte) =>
|
||||
byte.toString(16).padStart(2, '0')
|
||||
).join('')
|
||||
}
|
||||
|
||||
// Only the host document and its srcdoc child load, plus in-page anchors within them. A visual's
|
||||
// links reach the app through the bridge instead, so every other navigation is refused, not opened.
|
||||
function allowsLoad(request: ShouldStartLoadRequest): boolean {
|
||||
const url = request.url.split('#', 1)[0]
|
||||
return url === 'about:blank' || url === 'about:srcdoc'
|
||||
}
|
||||
|
||||
/**
|
||||
* One visual in a WebView: a trusted host document with the author HTML in an opaque sandboxed
|
||||
* child (see `buildMobileNativeChatVisualHostDocument`). The app accepts exactly two requests from
|
||||
* it, each token- and channel-checked here: a height, which the shared governor turns into the
|
||||
* frame's height, and an http(s) link to open.
|
||||
*/
|
||||
export const MobileNativeChatVisualFrame = memo(function MobileNativeChatVisualFrame({
|
||||
html,
|
||||
title,
|
||||
mode,
|
||||
onFailed
|
||||
}: MobileNativeChatVisualFrameProps) {
|
||||
const webView = useRef<WebView>(null)
|
||||
const [token] = useState(randomHex)
|
||||
const [generation, setGeneration] = useState(0)
|
||||
const [height, setHeight] = useState(MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT)
|
||||
const restarts = useRef(0)
|
||||
const lastLinkAt = useRef(-Infinity)
|
||||
|
||||
const built = useMemo(() => {
|
||||
const channel = randomHex()
|
||||
const visualDocument = buildNativeChatVisualDocument({
|
||||
html,
|
||||
channel,
|
||||
theme: MOBILE_NATIVE_CHAT_VISUAL_THEME
|
||||
})
|
||||
return {
|
||||
channel,
|
||||
source: {
|
||||
html: buildMobileNativeChatVisualHostDocument({
|
||||
visualDocument,
|
||||
channel,
|
||||
token,
|
||||
title,
|
||||
mode
|
||||
})
|
||||
}
|
||||
}
|
||||
}, [html, token, title, mode])
|
||||
|
||||
// A fresh governor per loaded document; reports queue behind a deferral rather than pile up.
|
||||
const sizing = useRef<{
|
||||
governor: ReturnType<typeof createNativeChatVisualHeightGovernor>
|
||||
latest: number | null
|
||||
deferred: ReturnType<typeof setTimeout> | null
|
||||
}>({ governor: createNativeChatVisualHeightGovernor(), latest: null, deferred: null })
|
||||
useEffect(() => {
|
||||
const state = sizing.current
|
||||
state.governor = createNativeChatVisualHeightGovernor()
|
||||
state.latest = null
|
||||
return () => {
|
||||
if (state.deferred) {
|
||||
clearTimeout(state.deferred)
|
||||
state.deferred = null
|
||||
}
|
||||
}
|
||||
}, [built, generation])
|
||||
|
||||
const applyHeight = useCallback(() => {
|
||||
const state = sizing.current
|
||||
state.deferred = null
|
||||
if (state.latest === null) {
|
||||
return
|
||||
}
|
||||
const decision = state.governor.decide(state.latest, Date.now())
|
||||
if (decision.kind === 'apply') {
|
||||
setHeight(decision.height)
|
||||
webView.current?.injectJavaScript(
|
||||
`window.${MOBILE_NATIVE_CHAT_VISUAL_APPLY_HEIGHT}(${decision.height}); true;`
|
||||
)
|
||||
} else if (decision.kind === 'defer') {
|
||||
state.deferred = setTimeout(applyHeight, decision.retryInMs)
|
||||
}
|
||||
}, [])
|
||||
|
||||
const onMessage = useCallback(
|
||||
(event: WebViewMessageEvent) => {
|
||||
const message = readMobileNativeChatVisualBridgeMessage(
|
||||
event.nativeEvent.data,
|
||||
token,
|
||||
built.channel
|
||||
)
|
||||
if (!message) {
|
||||
return
|
||||
}
|
||||
if (message.kind === 'escaped') {
|
||||
onFailed()
|
||||
return
|
||||
}
|
||||
if (message.kind === 'size') {
|
||||
if (mode === 'inline') {
|
||||
sizing.current.latest = message.height
|
||||
if (!sizing.current.deferred) {
|
||||
applyHeight()
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
const now = Date.now()
|
||||
if (now - lastLinkAt.current < MOBILE_NATIVE_CHAT_VISUAL_LINK_INTERVAL_MS) {
|
||||
return
|
||||
}
|
||||
lastLinkAt.current = now
|
||||
openExternalLink(message.url)
|
||||
},
|
||||
[token, built.channel, mode, applyHeight, onFailed]
|
||||
)
|
||||
|
||||
const restart = useCallback(() => {
|
||||
if (restarts.current >= MAX_PROCESS_RESTARTS) {
|
||||
onFailed()
|
||||
return
|
||||
}
|
||||
restarts.current += 1
|
||||
setHeight(MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT)
|
||||
setGeneration((value) => value + 1)
|
||||
}, [onFailed])
|
||||
|
||||
return (
|
||||
<WebView
|
||||
key={generation}
|
||||
ref={webView}
|
||||
source={built.source}
|
||||
style={mode === 'inline' ? [styles.inline, { height }] : styles.fullscreen}
|
||||
accessibilityLabel={title}
|
||||
// '*' so every navigation reaches `allowsLoad`: an origin outside this list is opened in the
|
||||
// system browser by the WebView library itself, with no gesture check.
|
||||
originWhitelist={['*']}
|
||||
onShouldStartLoadWithRequest={allowsLoad}
|
||||
javaScriptEnabled
|
||||
javaScriptCanOpenWindowsAutomatically={false}
|
||||
setSupportMultipleWindows={false}
|
||||
domStorageEnabled={false}
|
||||
allowFileAccess={false}
|
||||
allowsLinkPreview={false}
|
||||
mediaCapturePermissionGrantType="deny"
|
||||
mixedContentMode="never"
|
||||
// Android scales WebView text by the system font size; the page lays itself out.
|
||||
textZoom={100}
|
||||
scrollEnabled={false}
|
||||
bounces={false}
|
||||
showsVerticalScrollIndicator={false}
|
||||
showsHorizontalScrollIndicator={false}
|
||||
onMessage={onMessage}
|
||||
onError={onFailed}
|
||||
onHttpError={onFailed}
|
||||
onContentProcessDidTerminate={restart}
|
||||
onRenderProcessGone={restart}
|
||||
/>
|
||||
)
|
||||
})
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
inline: { width: '100%', backgroundColor: 'transparent' },
|
||||
fullscreen: { flex: 1, backgroundColor: 'transparent' }
|
||||
})
|
||||
@@ -0,0 +1,59 @@
|
||||
import { memo, useMemo } from 'react'
|
||||
import { StyleSheet, View } from 'react-native'
|
||||
import { buildNativeChatVisualDocument } from '../../../src/shared/native-chat-visual-shell'
|
||||
// The native component's own prop type, so a change to it fails here rather than drifting.
|
||||
import type { MobileNativeChatVisualFrameProps } from './MobileNativeChatVisualFrame'
|
||||
import { MOBILE_NATIVE_CHAT_VISUAL_THEME } from './mobile-native-chat-visual-theme'
|
||||
|
||||
/** No script runs, so nothing reports a height; a fixed frame the visual scrolls inside. */
|
||||
const SEALED_INLINE_HEIGHT = 320
|
||||
|
||||
/**
|
||||
* Web sibling, inside the hybrid shell's page: the visual rendered sealed, with no script.
|
||||
*
|
||||
* The page is served under the shell's response-header policy (`script-src 'self'`,
|
||||
* `frame-src 'none'`), and a `srcdoc` frame inherits that policy, so a visual's inline scripts
|
||||
* cannot run here without loosening the page's own policy. Static markup, CSS and images still
|
||||
* render; script-drawn charts do not. An empty `sandbox` keeps the frame an opaque origin that runs
|
||||
* nothing and navigates nothing, as the page's HTML preview does.
|
||||
*/
|
||||
export const MobileNativeChatVisualFrame = memo(function MobileNativeChatVisualFrame({
|
||||
html,
|
||||
title,
|
||||
mode
|
||||
}: MobileNativeChatVisualFrameProps) {
|
||||
// No script runs in this frame, so nothing ever reports on this channel.
|
||||
const document = useMemo(
|
||||
() =>
|
||||
buildNativeChatVisualDocument({
|
||||
html,
|
||||
channel: 'sealed',
|
||||
theme: MOBILE_NATIVE_CHAT_VISUAL_THEME
|
||||
}),
|
||||
[html]
|
||||
)
|
||||
return (
|
||||
<View style={mode === 'inline' ? styles.inline : styles.fullscreen}>
|
||||
<iframe
|
||||
title={title}
|
||||
sandbox=""
|
||||
srcDoc={document}
|
||||
referrerPolicy="no-referrer"
|
||||
style={IFRAME_STYLE}
|
||||
/>
|
||||
</View>
|
||||
)
|
||||
})
|
||||
|
||||
/** A DOM style, not a `StyleSheet` entry: this element is an `iframe` and not a react-native view. */
|
||||
const IFRAME_STYLE = {
|
||||
border: 'none',
|
||||
width: '100%',
|
||||
height: '100%',
|
||||
backgroundColor: 'transparent'
|
||||
} as const
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
inline: { width: '100%', height: SEALED_INLINE_HEIGHT },
|
||||
fullscreen: { flex: 1 }
|
||||
})
|
||||
@@ -66,6 +66,7 @@ vi.mock('lucide-react-native', () =>
|
||||
)
|
||||
)
|
||||
vi.mock('../MobileNativeChatMessage', () => ({ MobileNativeChatMessage: 'ChatMessage' }))
|
||||
vi.mock('../MobileNativeChatVisual', () => ({ useMobileNativeChatVisualRenderer: () => null }))
|
||||
vi.mock('../MobileNativeChatLiveLine', () => ({ MobileNativeChatLiveLine: 'LiveStatus' }))
|
||||
vi.mock('../MobileNativeChatAsk', () => ({ MobileNativeChatAsk: 'ChatAsk' }))
|
||||
vi.mock('../MobileAgentWorkingIndicator', () => ({
|
||||
|
||||
@@ -16,6 +16,7 @@ import type { MobileStructuredQueuedMessageControls } from './use-mobile-structu
|
||||
import type { MobileNativeChatPendingMessage } from './use-mobile-native-chat-drafts'
|
||||
import type { useMobileNativeChatSession } from './use-mobile-native-chat-session'
|
||||
import type { MobileNativeChatSessionOptionPickersProps } from './MobileNativeChatSessionOptionPickers'
|
||||
import type { MobileNativeChatVisualSource } from './mobile-native-chat-visual-read'
|
||||
|
||||
export type MobileNativeChatController = {
|
||||
/** Whether a tab's effective view is chat (per-tab override, else the default). */
|
||||
@@ -33,6 +34,8 @@ export type MobileNativeChatController = {
|
||||
nativeChatSession: ReturnType<typeof useMobileNativeChatSession>
|
||||
/** Structured lane: drives the per-turn status row and live tool progress. */
|
||||
nativeChatStructured: boolean
|
||||
/** Structured lane with a live client: where the transcript's visuals are read from. */
|
||||
nativeChatVisualSource: MobileNativeChatVisualSource | null
|
||||
nativeChatAgentWorking: boolean
|
||||
/** What labels the live turn's one indicator row; null off the structured lane. */
|
||||
nativeChatTurnIndicator: NativeChatLiveTurnIndicator | null
|
||||
|
||||
@@ -64,6 +64,7 @@ vi.mock('./MobileNativeChatMessage', () => ({ MobileNativeChatMessage: 'ChatMess
|
||||
vi.mock('./MobileNativeChatLiveLine', () => ({ MobileNativeChatLiveLine: 'LiveStatus' }))
|
||||
vi.mock('./MobileNativeChatComposer', () => ({ MobileNativeChatComposer: 'Composer' }))
|
||||
vi.mock('./MobileNativeChatQueuedMessages', () => ({ MobileNativeChatQueuedMessages: 'Queued' }))
|
||||
vi.mock('./MobileNativeChatVisual', () => ({ useMobileNativeChatVisualRenderer: () => null }))
|
||||
vi.mock('../components/ActionSheetModal', () => ({ ActionSheetModal: 'ActionSheetModal' }))
|
||||
vi.mock('./MobileNativeChatAsk', () => ({ MobileNativeChatAsk: 'ChatAsk' }))
|
||||
vi.mock('./MobileNativeChatPermission', () => ({ MobileNativeChatPermission: 'ChatPermission' }))
|
||||
|
||||
@@ -24,6 +24,14 @@ describe('nativeChatMessagePlainText', () => {
|
||||
).toBe(AGENT_SESSION_HOST_STATUS_COPY['history-item-too-large'])
|
||||
})
|
||||
|
||||
it('leaves a visual line out of the copied prose, but not one inside a code fence', () => {
|
||||
const text =
|
||||
'Chart:\n::orca-visual{file="usage.html"}\nDone.\n```\n::orca-visual{file="x.html"}\n```'
|
||||
expect(nativeChatMessagePlainText({ blocks: [{ type: 'text', text }] })).toBe(
|
||||
'Chart:\nDone.\n```\n::orca-visual{file="x.html"}\n```'
|
||||
)
|
||||
})
|
||||
|
||||
it('is empty for a message with no prose', () => {
|
||||
expect(
|
||||
nativeChatMessagePlainText({ blocks: [{ type: 'tool-call', name: 'Read', input: {} }] })
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
isAgentSessionHostStatusPresentation
|
||||
} from '../../../src/shared/agent-session-host-status-rows'
|
||||
import { isTextBlock } from '../../../src/shared/native-chat-types'
|
||||
import { withoutNativeChatVisualDirectiveLines } from '../../../src/shared/native-chat-visual-directive'
|
||||
import type { NativeChatMessage } from '../../../src/shared/native-chat-types'
|
||||
|
||||
/** Copy the displayed prose, excluding tool activity and image attachments. */
|
||||
@@ -13,7 +14,8 @@ export function nativeChatMessagePlainText(message: Pick<NativeChatMessage, 'blo
|
||||
.map((block) =>
|
||||
isAgentSessionHostStatusPresentation(block.presentation)
|
||||
? AGENT_SESSION_HOST_STATUS_COPY[block.presentation]
|
||||
: block.text
|
||||
: // A visual line is not prose; only the transcript can show it.
|
||||
withoutNativeChatVisualDirectiveLines(block.text)
|
||||
)
|
||||
.filter((text) => text.trim().length > 0)
|
||||
.join('\n\n')
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
import { createElement, type ReactNode } from 'react'
|
||||
import { act, create, type ReactTestRenderer } from 'react-test-renderer'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { NativeChatMessage } from '../../../src/shared/native-chat-types'
|
||||
import {
|
||||
MobileNativeChatVisualContext,
|
||||
type MobileNativeChatVisualRender
|
||||
} from './mobile-native-chat-visual-context'
|
||||
|
||||
vi.mock('react-native', async () => {
|
||||
const React = await import('react')
|
||||
const Text = ({ children, ...props }: { children?: ReactNode }): ReactNode =>
|
||||
React.createElement('Text', props, children)
|
||||
return {
|
||||
ActivityIndicator: 'ActivityIndicator',
|
||||
Animated: {
|
||||
Text,
|
||||
Value: class {
|
||||
setValue(): void {}
|
||||
},
|
||||
loop: (animation: unknown) => animation,
|
||||
sequence: () => ({ start: vi.fn(), stop: vi.fn() }),
|
||||
timing: () => ({ start: vi.fn(), stop: vi.fn() })
|
||||
},
|
||||
Image: 'Image',
|
||||
Platform: { OS: 'ios' },
|
||||
Pressable: 'Pressable',
|
||||
ScrollView: ({ children, ...props }: { children?: ReactNode }) =>
|
||||
React.createElement('ScrollView', props, children),
|
||||
Text,
|
||||
View: ({ children, ...props }: { children?: ReactNode }) =>
|
||||
React.createElement('View', props, children),
|
||||
StyleSheet: { create: (styles: unknown) => styles, hairlineWidth: 1 }
|
||||
}
|
||||
})
|
||||
vi.mock('expo-clipboard', () => ({ setStringAsync: vi.fn() }))
|
||||
vi.mock('lucide-react-native', () => ({
|
||||
ArrowUp: 'ArrowUp',
|
||||
Brain: 'Brain',
|
||||
ChevronDown: 'ChevronDown',
|
||||
Copy: 'Copy',
|
||||
SquareChevronRight: 'SquareChevronRight',
|
||||
SquareTerminal: 'SquareTerminal',
|
||||
Wrench: 'Wrench',
|
||||
ChevronRight: 'ChevronRight'
|
||||
}))
|
||||
vi.mock('../components/MobileMarkdown', () => ({ MobileMarkdown: 'MobileMarkdown' }))
|
||||
vi.mock('./MobileNativeChatMessageActionsSheet', () => ({
|
||||
MobileNativeChatMessageActionsSheet: 'MessageActionsSheet'
|
||||
}))
|
||||
|
||||
import { MobileNativeChatMessage } from './MobileNativeChatMessage'
|
||||
|
||||
const renderVisual: MobileNativeChatVisualRender = () => 'visual'
|
||||
|
||||
function message(id: string, role: NativeChatMessage['role'], text: string): NativeChatMessage {
|
||||
return { id, role, blocks: [{ type: 'text', text }], timestamp: null, source: 'transcript' }
|
||||
}
|
||||
|
||||
describe('MobileNativeChatMessage visuals', () => {
|
||||
let renderer: ReactTestRenderer | null = null
|
||||
|
||||
afterEach(() => {
|
||||
act(() => renderer?.unmount())
|
||||
renderer = null
|
||||
})
|
||||
|
||||
function markdownProps(
|
||||
row: NativeChatMessage,
|
||||
options: {
|
||||
visuals?: MobileNativeChatVisualRender | null
|
||||
activeTurnIsWorking?: boolean
|
||||
mayStillGrow?: boolean
|
||||
} = {}
|
||||
): Record<string, unknown> {
|
||||
act(() => {
|
||||
renderer = create(
|
||||
createElement(
|
||||
MobileNativeChatVisualContext.Provider,
|
||||
{ value: options.visuals === undefined ? renderVisual : options.visuals },
|
||||
createElement(MobileNativeChatMessage, {
|
||||
message: row,
|
||||
activeTurnIsWorking: options.activeTurnIsWorking,
|
||||
mayStillGrow: options.mayStillGrow ?? true
|
||||
})
|
||||
)
|
||||
)
|
||||
})
|
||||
return renderer!.root.find((node) => String(node.type) === 'MobileMarkdown').props
|
||||
}
|
||||
|
||||
it("renders a finished assistant reply's directives through the transcript renderer", () => {
|
||||
const props = markdownProps(message('a1', 'assistant', '::orca-visual{file="a.html"}'))
|
||||
expect(props.renderVisual).toBe(renderVisual)
|
||||
expect(props.content).toBe('::orca-visual{file="a.html"}')
|
||||
})
|
||||
|
||||
it('hides a directive still being typed while its turn works, and mounts finished lines', () => {
|
||||
const props = markdownProps(
|
||||
message(
|
||||
'a1',
|
||||
'assistant',
|
||||
'::orca-visual{file="done.html"}\nChart below.\n::orca-visual{file="usage"}'
|
||||
),
|
||||
{ activeTurnIsWorking: true }
|
||||
)
|
||||
expect(props.renderVisual).toBe(renderVisual)
|
||||
expect(props.content).toBe('::orca-visual{file="done.html"}\nChart below.\n')
|
||||
})
|
||||
|
||||
it('shows an earlier finished row of a working turn in full', () => {
|
||||
const text = 'Pick one:\n::orca-visual{file="options.html"}'
|
||||
expect(
|
||||
markdownProps(message('a0', 'assistant', text), {
|
||||
activeTurnIsWorking: true,
|
||||
mayStillGrow: false
|
||||
}).content
|
||||
).toBe(text)
|
||||
})
|
||||
|
||||
it('holds back only the last block of the row: text followed by a tool call is finished', () => {
|
||||
const text = 'Options:\n::orca-visual{file="options'
|
||||
const row: NativeChatMessage = {
|
||||
id: 'a1',
|
||||
role: 'assistant',
|
||||
blocks: [
|
||||
{ type: 'text', text },
|
||||
{ type: 'tool-call', name: 'AskUserQuestion', input: {} }
|
||||
],
|
||||
timestamp: null,
|
||||
source: 'transcript'
|
||||
}
|
||||
expect(markdownProps(row, { activeTurnIsWorking: true }).content).toBe(text)
|
||||
})
|
||||
|
||||
it('shows the whole reply once its turn settles', () => {
|
||||
const text = 'Chart below.\n::orca-visual{file="usage'
|
||||
expect(markdownProps(message('a1', 'assistant', text)).content).toBe(text)
|
||||
})
|
||||
|
||||
it('leaves directives as text where the chat has no visual source', () => {
|
||||
const props = markdownProps(message('a1', 'assistant', '::orca-visual{file="a.html"}'), {
|
||||
visuals: null,
|
||||
activeTurnIsWorking: true
|
||||
})
|
||||
expect(props.renderVisual).toBeUndefined()
|
||||
expect(props.content).toBe('::orca-visual{file="a.html"}')
|
||||
})
|
||||
|
||||
it('never renders visuals in system rows', () => {
|
||||
const props = markdownProps(message('s1', 'system', '::orca-visual{file="a.html"}'))
|
||||
expect(props.renderVisual).toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,104 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
NATIVE_CHAT_VISUAL_OPEN_LINK_TYPE,
|
||||
NATIVE_CHAT_VISUAL_SIZE_TYPE
|
||||
} from '../../../src/shared/native-chat-visual-shell'
|
||||
import { readMobileNativeChatVisualBridgeMessage } from './mobile-native-chat-visual-bridge'
|
||||
import { buildMobileNativeChatVisualHostDocument } from './mobile-native-chat-visual-host-document'
|
||||
|
||||
const TOKEN = 'f'.repeat(32)
|
||||
const CHANNEL = 'c'.repeat(32)
|
||||
|
||||
function frame(data: Record<string, unknown>, token: string = TOKEN): string {
|
||||
return JSON.stringify({ token, kind: 'frame', data: { channel: CHANNEL, ...data } })
|
||||
}
|
||||
|
||||
function read(raw: unknown) {
|
||||
return readMobileNativeChatVisualBridgeMessage(raw, TOKEN, CHANNEL)
|
||||
}
|
||||
|
||||
describe('readMobileNativeChatVisualBridgeMessage', () => {
|
||||
it('accepts a size report and an http(s) link relayed by the host document', () => {
|
||||
expect(read(frame({ type: NATIVE_CHAT_VISUAL_SIZE_TYPE, height: 312.4 }))).toEqual({
|
||||
kind: 'size',
|
||||
height: 312.4
|
||||
})
|
||||
expect(
|
||||
read(frame({ type: NATIVE_CHAT_VISUAL_OPEN_LINK_TYPE, url: 'https://example.com/a?b=c' }))
|
||||
).toEqual({ kind: 'open-link', url: 'https://example.com/a?b=c' })
|
||||
expect(read(JSON.stringify({ token: TOKEN, kind: 'escaped' }))).toEqual({ kind: 'escaped' })
|
||||
})
|
||||
|
||||
it('drops a message without this frame token or from another channel', () => {
|
||||
const size = { type: NATIVE_CHAT_VISUAL_SIZE_TYPE, height: 300 }
|
||||
expect(read(frame(size, 'other'))).toBeNull()
|
||||
expect(read(JSON.stringify({ kind: 'frame', data: { channel: CHANNEL, ...size } }))).toBeNull()
|
||||
expect(read(JSON.stringify({ kind: 'escaped' }))).toBeNull()
|
||||
expect(read(frame({ ...size, channel: 'another-visual' }))).toBeNull()
|
||||
})
|
||||
|
||||
it('drops links that are not plain http(s)', () => {
|
||||
for (const url of [
|
||||
'javascript:alert(1)',
|
||||
'file:///etc/passwd',
|
||||
'orca://open',
|
||||
'data:text/html,hi',
|
||||
`https://example.com/${'a'.repeat(3000)}`,
|
||||
42
|
||||
]) {
|
||||
expect(read(frame({ type: NATIVE_CHAT_VISUAL_OPEN_LINK_TYPE, url })), String(url)).toBeNull()
|
||||
}
|
||||
})
|
||||
|
||||
it('drops non-finite heights, unknown requests, non-strings, junk and oversized messages', () => {
|
||||
for (const height of [Number.NaN, Number.POSITIVE_INFINITY, -1, '300', null]) {
|
||||
expect(read(frame({ type: NATIVE_CHAT_VISUAL_SIZE_TYPE, height })), String(height)).toBeNull()
|
||||
}
|
||||
expect(read(frame({ type: 'orca-visual-navigate' }))).toBeNull()
|
||||
expect(read('not json')).toBeNull()
|
||||
expect(read(null)).toBeNull()
|
||||
expect(read(undefined)).toBeNull()
|
||||
expect(
|
||||
read(frame({ type: NATIVE_CHAT_VISUAL_SIZE_TYPE, height: 300, pad: 'x'.repeat(9000) }))
|
||||
).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildMobileNativeChatVisualHostDocument', () => {
|
||||
const build = (visualDocument: string) =>
|
||||
buildMobileNativeChatVisualHostDocument({
|
||||
visualDocument,
|
||||
channel: CHANNEL,
|
||||
token: TOKEN,
|
||||
title: 'Usage </script><script>alert(1)</script>',
|
||||
mode: 'inline'
|
||||
})
|
||||
|
||||
it('puts the visual in an opaque frame that may only run scripts', () => {
|
||||
const document = build('<p>hi</p>')
|
||||
expect(document).toContain("frame.setAttribute('sandbox', 'allow-scripts')")
|
||||
expect(document).not.toMatch(/allow-same-origin|allow-popups|allow-top-navigation|allow-forms/)
|
||||
expect(document.indexOf('Content-Security-Policy')).toBeLessThan(document.indexOf('<script>'))
|
||||
})
|
||||
|
||||
it('keeps the inline frame from scrolling so a drag on it scrolls the transcript', () => {
|
||||
expect(build('<p>hi</p>')).toContain("if (!C.fullscreen) frame.setAttribute('scrolling', 'no')")
|
||||
})
|
||||
|
||||
it('embeds the visual and title as script literals that cannot close the host script', () => {
|
||||
const document = build('</script><script>window.ReactNativeWebView.postMessage("x")</script>')
|
||||
// One script element: the host's own.
|
||||
expect(document.match(/<script>/g)).toHaveLength(1)
|
||||
expect(document.match(/<\/script>/g)).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('relays only the child window on its own channel, links only with focus and activation', () => {
|
||||
const document = build('<p>hi</p>')
|
||||
expect(document).toContain('event.source !== frame.contentWindow')
|
||||
expect(document).toContain('data.channel !== C.channel')
|
||||
expect(document).toContain(JSON.stringify(CHANNEL))
|
||||
expect(document).toContain('document.activeElement !== frame')
|
||||
expect(document).toContain('activation.isActive')
|
||||
expect(document).toContain(JSON.stringify(TOKEN))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,51 @@
|
||||
import {
|
||||
readNativeChatVisualFrameMessage,
|
||||
type NativeChatVisualFrameMessage
|
||||
} from '../../../src/shared/native-chat-visual-shell'
|
||||
|
||||
export type MobileNativeChatVisualBridgeEvent =
|
||||
| NativeChatVisualFrameMessage
|
||||
/** The visual navigated its frame away from its own document; the host document removed it. */
|
||||
| { kind: 'escaped' }
|
||||
|
||||
const MAX_BRIDGE_MESSAGE_LENGTH = 8 * 1024
|
||||
/** A link request is a deliberate tap; the app opens at most one per interval. */
|
||||
export const MOBILE_NATIVE_CHAT_VISUAL_LINK_INTERVAL_MS = 1_000
|
||||
|
||||
/**
|
||||
* The app's side of the visual bridge: a message counts only if it carries this frame's token
|
||||
* (which only the trusted host document holds) and is one of the two requests a visual may make,
|
||||
* from this visual's channel. Everything else, including a well-formed message without the token,
|
||||
* is dropped.
|
||||
*/
|
||||
export function readMobileNativeChatVisualBridgeMessage(
|
||||
raw: unknown,
|
||||
token: string,
|
||||
channel: string
|
||||
): MobileNativeChatVisualBridgeEvent | null {
|
||||
if (typeof raw !== 'string' || raw.length > MAX_BRIDGE_MESSAGE_LENGTH) {
|
||||
return null
|
||||
}
|
||||
let message: unknown
|
||||
try {
|
||||
message = JSON.parse(raw)
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
if (
|
||||
typeof message !== 'object' ||
|
||||
message === null ||
|
||||
!('token' in message) ||
|
||||
message.token !== token ||
|
||||
!('kind' in message)
|
||||
) {
|
||||
return null
|
||||
}
|
||||
if (message.kind === 'escaped') {
|
||||
return { kind: 'escaped' }
|
||||
}
|
||||
if (message.kind === 'frame' && 'data' in message) {
|
||||
return readNativeChatVisualFrameMessage(message.data, channel)
|
||||
}
|
||||
return null
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { createContext, type ReactNode } from 'react'
|
||||
import type { NativeChatVisualDirective } from '../../../src/shared/native-chat-visual-directive'
|
||||
|
||||
export type MobileNativeChatVisualRender = (
|
||||
directive: NativeChatVisualDirective,
|
||||
index: number
|
||||
) => ReactNode
|
||||
|
||||
/**
|
||||
* How this transcript renders `::orca-visual` lines, or null where it cannot (no structured chat or
|
||||
* no client). Rows read only this, so they never import the WebView frame themselves.
|
||||
*/
|
||||
export const MobileNativeChatVisualContext = createContext<MobileNativeChatVisualRender | null>(
|
||||
null
|
||||
)
|
||||
@@ -0,0 +1,132 @@
|
||||
import {
|
||||
NATIVE_CHAT_VISUAL_CSP,
|
||||
NATIVE_CHAT_VISUAL_OPEN_LINK_TYPE,
|
||||
NATIVE_CHAT_VISUAL_SIZE_TYPE
|
||||
} from '../../../src/shared/native-chat-visual-shell'
|
||||
import { inlineScriptLiteral } from '../components/inline-script-json'
|
||||
|
||||
/** `inline` sizes the frame to the height the app applies; `fullscreen` fills the screen and scrolls. */
|
||||
export type MobileNativeChatVisualHostMode = 'inline' | 'fullscreen'
|
||||
|
||||
export const MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT = 160
|
||||
|
||||
/** The host page's function the app calls to size the frame to a height it has decided. */
|
||||
export const MOBILE_NATIVE_CHAT_VISUAL_APPLY_HEIGHT = '__orcaVisualApplyHeight'
|
||||
|
||||
// The host forwards at most one size report per interval (the latest wins) and one link per window.
|
||||
const SIZE_FORWARD_INTERVAL_MS = 100
|
||||
const LINK_WINDOW_MS = 5_000
|
||||
|
||||
/**
|
||||
* The trusted document the WebView loads. A native WebView's top document gets no `sandbox`, so
|
||||
* the author HTML never runs here: it runs in an opaque `sandbox="allow-scripts"` srcdoc child, and
|
||||
* this document is the only one the app listens to (the WebView patch drops messages from any frame
|
||||
* but the main one).
|
||||
*
|
||||
* - It relays only messages whose `source` is the child's window, as data for the app to validate,
|
||||
* and stamps them with `token`, which the child never sees.
|
||||
* - A link request is relayed only while the child frame holds focus and the page holds user
|
||||
* activation, and at most once per activation window.
|
||||
* - A second `load` of the child means it navigated away from its document: the frame is removed
|
||||
* and the app told, so a replacement document never inherits the frame.
|
||||
* - It never sizes the frame from a report itself; the app decides heights and calls back.
|
||||
* - Inline, the frame does not scroll, so a drag that starts on it scrolls the transcript.
|
||||
*
|
||||
* The child inherits this document's policy (a srcdoc frame has no URL of its own) and adds the
|
||||
* same policy from its own meta, so this document carries the visual policy too.
|
||||
*/
|
||||
export function buildMobileNativeChatVisualHostDocument(input: {
|
||||
visualDocument: string
|
||||
/** The channel the visual document was built with; only messages on it are relayed. */
|
||||
channel: string
|
||||
token: string
|
||||
title: string
|
||||
mode: MobileNativeChatVisualHostMode
|
||||
}): string {
|
||||
const frameHeight =
|
||||
input.mode === 'fullscreen' ? '100vh' : `${MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT}px`
|
||||
const constants = inlineScriptLiteral({
|
||||
token: input.token,
|
||||
channel: input.channel,
|
||||
fullscreen: input.mode === 'fullscreen',
|
||||
title: input.title,
|
||||
size: NATIVE_CHAT_VISUAL_SIZE_TYPE,
|
||||
link: NATIVE_CHAT_VISUAL_OPEN_LINK_TYPE,
|
||||
applyHeight: MOBILE_NATIVE_CHAT_VISUAL_APPLY_HEIGHT,
|
||||
sizeIntervalMs: SIZE_FORWARD_INTERVAL_MS,
|
||||
linkWindowMs: LINK_WINDOW_MS
|
||||
})
|
||||
return `<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="Content-Security-Policy" content="${NATIVE_CHAT_VISUAL_CSP}">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
|
||||
<style>html,body{margin:0;padding:0;background:transparent;overflow:hidden}iframe{display:block;border:0;width:100%;height:${frameHeight};background:transparent}</style>
|
||||
</head>
|
||||
<body>
|
||||
<script>
|
||||
(function () {
|
||||
'use strict'
|
||||
var C = ${constants}
|
||||
var channel = window.ReactNativeWebView
|
||||
function send(message) {
|
||||
message.token = C.token
|
||||
if (channel) channel.postMessage(JSON.stringify(message))
|
||||
}
|
||||
var frame = document.createElement('iframe')
|
||||
frame.setAttribute('sandbox', 'allow-scripts')
|
||||
frame.setAttribute('referrerpolicy', 'no-referrer')
|
||||
frame.setAttribute('allow', "camera 'none'; microphone 'none'; geolocation 'none'; clipboard-read 'none'; clipboard-write 'none'; display-capture 'none'")
|
||||
frame.setAttribute('title', C.title)
|
||||
// iOS gives a scrollable frame its own scroll view, which takes a drag meant for the transcript;
|
||||
// inline the frame is sized to its content (full screen still scrolls).
|
||||
if (!C.fullscreen) frame.setAttribute('scrolling', 'no')
|
||||
var loads = 0
|
||||
frame.addEventListener('load', function () {
|
||||
loads += 1
|
||||
if (loads > 1) {
|
||||
frame.remove()
|
||||
send({ kind: 'escaped' })
|
||||
}
|
||||
})
|
||||
window[C.applyHeight] = function (height) {
|
||||
if (!C.fullscreen && typeof height === 'number' && isFinite(height)) frame.style.height = height + 'px'
|
||||
}
|
||||
var pendingSize = null
|
||||
var sizeTimer = null
|
||||
function flushSize() {
|
||||
sizeTimer = null
|
||||
if (pendingSize === null) return
|
||||
send({ kind: 'frame', data: pendingSize })
|
||||
pendingSize = null
|
||||
}
|
||||
var lastLinkAt = -Infinity
|
||||
window.addEventListener('message', function (event) {
|
||||
if (!frame.contentWindow || event.source !== frame.contentWindow) return
|
||||
var data = event.data
|
||||
// The visual chooses every field, so only an exact channel match is relayed, as the host's own copy.
|
||||
if (!data || typeof data !== 'object' || data.channel !== C.channel) return
|
||||
if (data.type === C.size) {
|
||||
if (C.fullscreen) return
|
||||
pendingSize = { type: C.size, channel: C.channel, height: Number(data.height) }
|
||||
if (sizeTimer === null) sizeTimer = setTimeout(flushSize, C.sizeIntervalMs)
|
||||
return
|
||||
}
|
||||
if (data.type === C.link) {
|
||||
var activation = navigator.userActivation
|
||||
var now = Date.now()
|
||||
if (document.activeElement !== frame || !activation || !activation.isActive) return
|
||||
if (typeof data.url !== 'string' || data.url.length > 4096) return
|
||||
if (now - lastLinkAt < C.linkWindowMs) return
|
||||
lastLinkAt = now
|
||||
send({ kind: 'frame', data: { type: C.link, channel: C.channel, url: data.url } })
|
||||
}
|
||||
})
|
||||
frame.srcdoc = ${inlineScriptLiteral(input.visualDocument)}
|
||||
document.body.appendChild(frame)
|
||||
})()
|
||||
</script>
|
||||
</body>
|
||||
</html>`
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { RpcClient } from '../transport/rpc-client'
|
||||
import type { RpcResponse } from '../transport/types'
|
||||
import {
|
||||
cachedMobileNativeChatVisual,
|
||||
readMobileNativeChatVisual,
|
||||
resetMobileNativeChatVisualCacheForTest,
|
||||
type MobileNativeChatVisualSource
|
||||
} from './mobile-native-chat-visual-read'
|
||||
|
||||
const REVISION_A = 'a'.repeat(32)
|
||||
const REVISION_B = 'b'.repeat(32)
|
||||
|
||||
function success(result: unknown): RpcResponse {
|
||||
return { id: '1', ok: true, result }
|
||||
}
|
||||
|
||||
function failure(code: string): RpcResponse {
|
||||
return { id: '1', ok: false, error: { code, message: code } }
|
||||
}
|
||||
|
||||
function sourceWith(sendRequest: ReturnType<typeof vi.fn>): MobileNativeChatVisualSource {
|
||||
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the reader calls only sendRequest; the rest of RpcClient is connection lifecycle it never touches.
|
||||
const client = { sendRequest } as unknown as RpcClient
|
||||
return { client, sessionId: 'session-1' }
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
resetMobileNativeChatVisualCacheForTest()
|
||||
})
|
||||
|
||||
describe('readMobileNativeChatVisual', () => {
|
||||
it('reads the visual by session and bare file name, then caches it', async () => {
|
||||
const sendRequest = vi
|
||||
.fn()
|
||||
.mockResolvedValue(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' }))
|
||||
const source = sourceWith(sendRequest)
|
||||
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'ready',
|
||||
html: '<p>',
|
||||
revision: REVISION_A
|
||||
})
|
||||
expect(sendRequest).toHaveBeenCalledWith(
|
||||
'agentSession.readVisual',
|
||||
{ sessionId: 'session-1', file: 'chart.html' },
|
||||
expect.objectContaining({ timeoutMs: expect.any(Number) })
|
||||
)
|
||||
expect(cachedMobileNativeChatVisual(source, 'chart.html')).toEqual({
|
||||
html: '<p>',
|
||||
revision: REVISION_A
|
||||
})
|
||||
})
|
||||
|
||||
it('revalidates with the held revision and keeps the cached bytes on unchanged', async () => {
|
||||
const sendRequest = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' }))
|
||||
.mockResolvedValueOnce(
|
||||
success({ ok: true, revision: REVISION_A, sizeBytes: 3, unchanged: true })
|
||||
)
|
||||
const source = sourceWith(sendRequest)
|
||||
await readMobileNativeChatVisual(source, 'chart.html')
|
||||
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'ready',
|
||||
html: '<p>',
|
||||
revision: REVISION_A
|
||||
})
|
||||
expect(sendRequest.mock.calls[1]?.[1]).toEqual({
|
||||
sessionId: 'session-1',
|
||||
file: 'chart.html',
|
||||
knownRevision: REVISION_A
|
||||
})
|
||||
})
|
||||
|
||||
it('replaces the cached bytes when the host has a new revision', async () => {
|
||||
const sendRequest = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' }))
|
||||
.mockResolvedValueOnce(
|
||||
success({ ok: true, revision: REVISION_B, sizeBytes: 4, html: '<hr>' })
|
||||
)
|
||||
const source = sourceWith(sendRequest)
|
||||
await readMobileNativeChatVisual(source, 'chart.html')
|
||||
await readMobileNativeChatVisual(source, 'chart.html')
|
||||
expect(cachedMobileNativeChatVisual(source, 'chart.html')?.revision).toBe(REVISION_B)
|
||||
})
|
||||
|
||||
it('drops the cached bytes when the host refuses a visual it served before', async () => {
|
||||
const source = sourceWith(
|
||||
vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(
|
||||
success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' })
|
||||
)
|
||||
.mockResolvedValueOnce(success({ ok: false, error: 'not_found' }))
|
||||
)
|
||||
await readMobileNativeChatVisual(source, 'chart.html')
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'refused'
|
||||
})
|
||||
expect(cachedMobileNativeChatVisual(source, 'chart.html')).toBeNull()
|
||||
})
|
||||
|
||||
it('reads a host verdict and an older host as refused', async () => {
|
||||
for (const response of [
|
||||
success({ ok: false, error: 'not_found' }),
|
||||
success({ ok: false, error: 'some_future_error' }),
|
||||
failure('method_not_found'),
|
||||
failure('forbidden')
|
||||
]) {
|
||||
resetMobileNativeChatVisualCacheForTest()
|
||||
const source = sourceWith(vi.fn().mockResolvedValue(response))
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'refused'
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
it('reads an error reply and a malformed reply as no verdict, keeping the cache', async () => {
|
||||
for (const response of [
|
||||
failure('runtime_error'),
|
||||
success({ ok: true, revision: 'not-hex', html: '' }),
|
||||
success('<html>')
|
||||
]) {
|
||||
resetMobileNativeChatVisualCacheForTest()
|
||||
const sendRequest = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(
|
||||
success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' })
|
||||
)
|
||||
.mockResolvedValueOnce(response)
|
||||
const source = sourceWith(sendRequest)
|
||||
await readMobileNativeChatVisual(source, 'chart.html')
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'unreachable'
|
||||
})
|
||||
expect(cachedMobileNativeChatVisual(source, 'chart.html')).not.toBeNull()
|
||||
}
|
||||
})
|
||||
|
||||
it('reads a transport failure as unreachable, never as refused', async () => {
|
||||
const source = sourceWith(vi.fn().mockRejectedValue(new Error('socket closed')))
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'unreachable'
|
||||
})
|
||||
})
|
||||
|
||||
it('treats an unchanged answer for a revision it never sent as unreachable and drops the entry', async () => {
|
||||
const source = sourceWith(
|
||||
vi.fn().mockResolvedValue(success({ ok: true, revision: REVISION_A, unchanged: true }))
|
||||
)
|
||||
await expect(readMobileNativeChatVisual(source, 'chart.html')).resolves.toEqual({
|
||||
kind: 'unreachable'
|
||||
})
|
||||
expect(cachedMobileNativeChatVisual(source, 'chart.html')).toBeNull()
|
||||
})
|
||||
|
||||
it('shares one request between concurrent reads of the same visual', async () => {
|
||||
let resolve: (response: RpcResponse) => void = () => {}
|
||||
const sendRequest = vi.fn(
|
||||
() =>
|
||||
new Promise<RpcResponse>((settle) => {
|
||||
resolve = settle
|
||||
})
|
||||
)
|
||||
const source = sourceWith(sendRequest)
|
||||
const first = readMobileNativeChatVisual(source, 'chart.html')
|
||||
const second = readMobileNativeChatVisual(source, 'chart.html')
|
||||
resolve(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' }))
|
||||
await expect(Promise.all([first, second])).resolves.toHaveLength(2)
|
||||
expect(sendRequest).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('keys the cache by client and session, so another host or chat never sees this one', async () => {
|
||||
const source = sourceWith(
|
||||
vi
|
||||
.fn()
|
||||
.mockResolvedValue(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' }))
|
||||
)
|
||||
await readMobileNativeChatVisual(source, 'chart.html')
|
||||
expect(cachedMobileNativeChatVisual({ ...source, sessionId: 'other' }, 'chart.html')).toBeNull()
|
||||
const otherHost = sourceWith(vi.fn()).client
|
||||
expect(cachedMobileNativeChatVisual({ ...source, client: otherHost }, 'chart.html')).toBeNull()
|
||||
})
|
||||
|
||||
it('evicts the oldest visuals past the entry bound', async () => {
|
||||
const source = sourceWith(
|
||||
vi
|
||||
.fn()
|
||||
.mockResolvedValue(success({ ok: true, revision: REVISION_A, sizeBytes: 3, html: '<p>' }))
|
||||
)
|
||||
for (let index = 0; index < 17; index += 1) {
|
||||
await readMobileNativeChatVisual(source, `v${index}.html`)
|
||||
}
|
||||
expect(cachedMobileNativeChatVisual(source, 'v0.html')).toBeNull()
|
||||
expect(cachedMobileNativeChatVisual(source, 'v16.html')).not.toBeNull()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,169 @@
|
||||
import { z } from 'zod'
|
||||
import { NATIVE_CHAT_VISUAL_MAX_BYTES } from '../../../src/shared/native-chat-visual-directive'
|
||||
import { bindDeferredRpcOperation, defineRpcOperation } from '../transport/rpc-operation'
|
||||
import { rpcResultVariant } from '../transport/rpc-operation-result-reader'
|
||||
import { isMobileMethodUnavailableError } from '../transport/mobile-method-unavailable'
|
||||
import type { RpcClient } from '../transport/rpc-client'
|
||||
|
||||
/** Where a chat's visuals are read from: the host that owns the structured session. */
|
||||
export type MobileNativeChatVisualSource = {
|
||||
client: RpcClient
|
||||
sessionId: string
|
||||
}
|
||||
|
||||
export type MobileNativeChatVisualRead =
|
||||
| { kind: 'ready'; html: string; revision: string }
|
||||
/** The host positively refused (missing, too large, outside the folder...) or cannot serve visuals. */
|
||||
| { kind: 'refused' }
|
||||
/** No verdict: no answer, an error reply, an unreadable answer. Worth asking again. */
|
||||
| { kind: 'unreachable' }
|
||||
|
||||
const Revision = z.string().regex(/^[0-9a-f]{16,64}$/)
|
||||
|
||||
const visualReadReplySchema = z.union([
|
||||
z.object({
|
||||
ok: z.literal(true),
|
||||
revision: Revision,
|
||||
html: z.string().max(NATIVE_CHAT_VISUAL_MAX_BYTES)
|
||||
}),
|
||||
z.object({ ok: z.literal(true), revision: Revision, unchanged: z.literal(true) }),
|
||||
// A newer host's error code still reads as a refusal.
|
||||
z.object({ ok: z.literal(false), error: z.string() })
|
||||
])
|
||||
|
||||
/** An error reply and an unreadable reply both read as null. */
|
||||
const nativeChatVisualRead = bindDeferredRpcOperation(
|
||||
defineRpcOperation({
|
||||
name: 'agentSession.read-visual',
|
||||
method: 'agentSession.readVisual',
|
||||
acceptance: 'object-result-or-null',
|
||||
barrier: 'after-caller-barrier',
|
||||
read: rpcResultVariant('native-chat-visual', visualReadReplySchema)
|
||||
})
|
||||
)
|
||||
|
||||
const READ_TIMEOUT_MS = 15_000
|
||||
// Each entry is at most 512 KiB of UTF-8; the character bound is what caps memory.
|
||||
const MAX_CACHE_ENTRIES = 16
|
||||
const MAX_CACHE_CHARS = 4 * 1024 * 1024
|
||||
|
||||
type CachedVisual = { html: string; revision: string }
|
||||
|
||||
const cache = new Map<string, CachedVisual>()
|
||||
const inFlight = new Map<string, Promise<MobileNativeChatVisualRead>>()
|
||||
|
||||
// The client stands for the host: each paired host has its own, so one host's visuals never
|
||||
// answer for another's even if two session ids collided.
|
||||
const clientKeys = new WeakMap<RpcClient, number>()
|
||||
let nextClientKey = 0
|
||||
|
||||
function cacheKey(source: MobileNativeChatVisualSource, file: string): string {
|
||||
let clientKey = clientKeys.get(source.client)
|
||||
if (clientKey === undefined) {
|
||||
nextClientKey += 1
|
||||
clientKey = nextClientKey
|
||||
clientKeys.set(source.client, clientKey)
|
||||
}
|
||||
return JSON.stringify([clientKey, source.sessionId, file])
|
||||
}
|
||||
|
||||
function remember(key: string, entry: CachedVisual): void {
|
||||
cache.delete(key)
|
||||
cache.set(key, entry)
|
||||
let chars = 0
|
||||
for (const value of cache.values()) {
|
||||
chars += value.html.length
|
||||
}
|
||||
// Oldest first: Map iteration follows insertion order, and a hit re-inserts.
|
||||
for (const [oldKey, value] of cache) {
|
||||
if (cache.size <= MAX_CACHE_ENTRIES && chars <= MAX_CACHE_CHARS) {
|
||||
break
|
||||
}
|
||||
if (oldKey === key) {
|
||||
continue
|
||||
}
|
||||
cache.delete(oldKey)
|
||||
chars -= value.html.length
|
||||
}
|
||||
}
|
||||
|
||||
/** What this phone already holds for a visual, to paint before revalidating. */
|
||||
export function cachedMobileNativeChatVisual(
|
||||
source: MobileNativeChatVisualSource,
|
||||
file: string
|
||||
): CachedVisual | null {
|
||||
return cache.get(cacheKey(source, file)) ?? null
|
||||
}
|
||||
|
||||
async function readOnce(
|
||||
source: MobileNativeChatVisualSource,
|
||||
file: string,
|
||||
key: string
|
||||
): Promise<MobileNativeChatVisualRead> {
|
||||
const known = cache.get(key)
|
||||
let response
|
||||
try {
|
||||
response = await nativeChatVisualRead.request(
|
||||
source.client,
|
||||
{
|
||||
sessionId: source.sessionId,
|
||||
file,
|
||||
...(known ? { knownRevision: known.revision } : {})
|
||||
},
|
||||
{ timeoutMs: READ_TIMEOUT_MS, budgetSpansConnect: true }
|
||||
)
|
||||
} catch {
|
||||
return { kind: 'unreachable' }
|
||||
}
|
||||
if (!response.ok && isMobileMethodUnavailableError(response.error.code, response.error.message)) {
|
||||
// An older host (its mobile allowlist answers `forbidden`): asking again will not help.
|
||||
return { kind: 'refused' }
|
||||
}
|
||||
const reply = nativeChatVisualRead.interpret(response)
|
||||
if (reply === null) {
|
||||
// An error reply, an older host or an unreadable answer is not a verdict on the file.
|
||||
return { kind: 'unreachable' }
|
||||
}
|
||||
if (!reply.ok) {
|
||||
// The host no longer serves it (deleted, now too large...): stop painting the old bytes.
|
||||
cache.delete(key)
|
||||
return { kind: 'refused' }
|
||||
}
|
||||
if ('html' in reply) {
|
||||
remember(key, { html: reply.html, revision: reply.revision })
|
||||
return { kind: 'ready', html: reply.html, revision: reply.revision }
|
||||
}
|
||||
// `unchanged` answers the revision this phone sent; anything else is a host that lost track.
|
||||
if (known && known.revision === reply.revision) {
|
||||
remember(key, known)
|
||||
return { kind: 'ready', html: known.html, revision: known.revision }
|
||||
}
|
||||
cache.delete(key)
|
||||
return { kind: 'unreachable' }
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads one visual from the chat's owning host, revalidating what the phone already holds by
|
||||
* revision. Concurrent reads of the same visual share one request.
|
||||
*/
|
||||
export function readMobileNativeChatVisual(
|
||||
source: MobileNativeChatVisualSource,
|
||||
file: string
|
||||
): Promise<MobileNativeChatVisualRead> {
|
||||
const key = cacheKey(source, file)
|
||||
const pending = inFlight.get(key)
|
||||
if (pending) {
|
||||
return pending
|
||||
}
|
||||
const read = readOnce(source, file, key).finally(() => {
|
||||
inFlight.delete(key)
|
||||
})
|
||||
inFlight.set(key, read)
|
||||
return read
|
||||
}
|
||||
|
||||
/** Test seam: forget every cached visual. */
|
||||
export function resetMobileNativeChatVisualCacheForTest(): void {
|
||||
cache.clear()
|
||||
inFlight.clear()
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import type { NativeChatVisualTheme } from '../../../src/shared/native-chat-visual-shell'
|
||||
import { colors, radii } from '../theme/mobile-theme'
|
||||
|
||||
/**
|
||||
* The phone's one (dark) theme, as the variables a visual styles against. The app has no theme
|
||||
* switch, so a visual receives this before its content runs and never needs a live update.
|
||||
* Chart colors are desktop's dark `--chart-1..5` (the blue 300-800 ramp), as hex.
|
||||
*/
|
||||
export const MOBILE_NATIVE_CHAT_VISUAL_THEME: NativeChatVisualTheme = {
|
||||
colorScheme: 'dark',
|
||||
tokens: {
|
||||
'--background': colors.bgBase,
|
||||
'--foreground': colors.textPrimary,
|
||||
'--card': colors.bgPanel,
|
||||
'--card-foreground': colors.textPrimary,
|
||||
'--popover': colors.bgRaised,
|
||||
'--popover-foreground': colors.textPrimary,
|
||||
'--primary': colors.surfaceBright,
|
||||
'--primary-foreground': colors.bgBase,
|
||||
'--secondary': colors.bgRaised,
|
||||
'--secondary-foreground': colors.textPrimary,
|
||||
'--muted': colors.bgRaised,
|
||||
'--muted-foreground': colors.textMuted,
|
||||
'--accent': colors.bgRaised,
|
||||
'--accent-foreground': colors.textPrimary,
|
||||
'--destructive': colors.statusRed,
|
||||
'--destructive-foreground': colors.onAccent,
|
||||
'--border': colors.borderSubtle,
|
||||
'--input': colors.borderSubtle,
|
||||
'--ring': colors.textMuted,
|
||||
'--radius': `${radii.row}px`,
|
||||
'--chart-1': '#8ec5ff',
|
||||
'--chart-2': '#2b7fff',
|
||||
'--chart-3': '#155dfc',
|
||||
'--chart-4': '#1447e6',
|
||||
'--chart-5': '#193cb8',
|
||||
'--font-sans': '-apple-system, system-ui, Roboto, sans-serif',
|
||||
'--font-mono': 'Menlo, monospace'
|
||||
}
|
||||
}
|
||||
@@ -298,6 +298,7 @@ export function useMobileNativeChatController(args: {
|
||||
nativeChatSession,
|
||||
/** Structured lane: drives the per-turn status row and live tool progress. */
|
||||
nativeChatStructured: activeChatStructured,
|
||||
nativeChatVisualSource: activeChatStructured ? structuredNativeChat.visualSource : null,
|
||||
nativeChatAgentWorking,
|
||||
nativeChatTurnIndicator: activeChatStructured ? structuredNativeChat.turnIndicator : null,
|
||||
nativeChatWorkingStartedAt: activeChatStructured ? structuredNativeChat.workingStartedAt : null,
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
import { createElement } from 'react'
|
||||
import { act, create, type ReactTestRenderer } from 'react-test-renderer'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import type { NativeChatMessage } from '../../../src/shared/native-chat-types'
|
||||
import { Harness, Result, userMessage } from './use-mobile-native-chat-turn-disclosure.test-fixture'
|
||||
|
||||
function assistantMessage(id: string): NativeChatMessage {
|
||||
return {
|
||||
id,
|
||||
role: 'assistant',
|
||||
blocks: [{ type: 'text', text: id }],
|
||||
timestamp: null,
|
||||
source: 'transcript'
|
||||
}
|
||||
}
|
||||
|
||||
describe('useMobileNativeChatTurnDisclosure growing row', () => {
|
||||
let renderer: ReactTestRenderer | null = null
|
||||
|
||||
afterEach(() => {
|
||||
act(() => renderer?.unmount())
|
||||
renderer = null
|
||||
})
|
||||
|
||||
it('marks no row while a prompt is open', () => {
|
||||
const messages = [userMessage('u1'), assistantMessage('a1')]
|
||||
act(() => {
|
||||
renderer = create(createElement(Harness, { messages, enabled: true, lineYields: true }))
|
||||
})
|
||||
const { disclosure } = renderer!.root.findByType(Result).props
|
||||
expect(disclosure.resolveRow(1, messages[1]).mayStillGrow).toBe(false)
|
||||
})
|
||||
|
||||
it('marks only the newest assistant row, even when a user row follows it', () => {
|
||||
const messages = [
|
||||
userMessage('u1'),
|
||||
assistantMessage('a1'),
|
||||
assistantMessage('a2'),
|
||||
userMessage('u2')
|
||||
]
|
||||
act(() => {
|
||||
renderer = create(createElement(Harness, { messages, enabled: true }))
|
||||
})
|
||||
const { disclosure } = renderer!.root.findByType(Result).props
|
||||
expect(
|
||||
messages.map((message, index) => disclosure.resolveRow(index, message).mayStillGrow)
|
||||
).toEqual([false, false, true, false])
|
||||
})
|
||||
})
|
||||
@@ -33,6 +33,8 @@ export type MobileNativeChatTurnRow = {
|
||||
/** Set only on a settled turn — the one row that has activity to disclose. */
|
||||
turnKey?: string
|
||||
activeTurnIsWorking: boolean
|
||||
/** The newest assistant row of a live turn with no prompt open: its last text may still grow. */
|
||||
mayStillGrow: boolean
|
||||
/** The live activity line discloses this open reasoning block, so its row draws nothing. */
|
||||
reasoningIsLive: boolean
|
||||
/** A reasoning row's disclosure, keyed like the live line's so an opened block stays open. */
|
||||
@@ -195,6 +197,10 @@ export function useMobileNativeChatTurnDisclosure({
|
||||
},
|
||||
[expandedReasoning, line]
|
||||
)
|
||||
const latestAssistantId = useMemo(
|
||||
() => waiting.listMessages.findLast((row) => row.role === 'assistant')?.id ?? null,
|
||||
[waiting.listMessages]
|
||||
)
|
||||
const resolveRow = useCallback(
|
||||
(listIndex: number, message: NativeChatMessage): MobileNativeChatTurnRow => {
|
||||
const index = waiting.indexById?.get(message.id) ?? listIndex
|
||||
@@ -219,6 +225,7 @@ export function useMobileNativeChatTurnDisclosure({
|
||||
turnKey: turnKey && turnStatus?.workedSeconds != null ? turnKey : undefined,
|
||||
// With no user boundary at all, the session's working state stays authoritative.
|
||||
activeTurnIsWorking: inLiveWorkingTurn(index),
|
||||
mayStillGrow: !lineYields && message.id === latestAssistantId,
|
||||
reasoningIsLive: message.id === liveReasoningId,
|
||||
reasoningExpanded:
|
||||
message.role === 'reasoning' &&
|
||||
@@ -236,6 +243,8 @@ export function useMobileNativeChatTurnDisclosure({
|
||||
completedByTurn,
|
||||
expandedTurnIds,
|
||||
inLiveWorkingTurn,
|
||||
latestAssistantId,
|
||||
lineYields,
|
||||
liveReasoningId,
|
||||
expandedReasoning,
|
||||
toggleReasoning
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
import { createElement } from 'react'
|
||||
import { act, create, type ReactTestRenderer } from 'react-test-renderer'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { RpcClient } from '../transport/rpc-client'
|
||||
import type { MobileNativeChatVisualRead } from './mobile-native-chat-visual-read'
|
||||
|
||||
const reads = vi.hoisted(() => {
|
||||
const next: MobileNativeChatVisualRead[] = []
|
||||
return { next, calls: 0 }
|
||||
})
|
||||
|
||||
vi.mock('./mobile-native-chat-visual-read', () => ({
|
||||
cachedMobileNativeChatVisual: () => null,
|
||||
readMobileNativeChatVisual: () => {
|
||||
reads.calls += 1
|
||||
return Promise.resolve(reads.next.shift() ?? { kind: 'unreachable' })
|
||||
}
|
||||
}))
|
||||
|
||||
import {
|
||||
useMobileNativeChatVisual,
|
||||
type MobileNativeChatVisualState
|
||||
} from './use-mobile-native-chat-visual'
|
||||
|
||||
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the read module is mocked; the client is only an identity here.
|
||||
const source = { client: {} as RpcClient, sessionId: 'session-1' }
|
||||
|
||||
let latest: { state: MobileNativeChatVisualState; retry: () => void } | null = null
|
||||
function Probe() {
|
||||
latest = useMobileNativeChatVisual(source, 'chart.html')
|
||||
return null
|
||||
}
|
||||
|
||||
describe('useMobileNativeChatVisual', () => {
|
||||
let renderer: ReactTestRenderer | null = null
|
||||
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers()
|
||||
reads.next = []
|
||||
reads.calls = 0
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
act(() => renderer?.unmount())
|
||||
renderer = null
|
||||
latest = null
|
||||
vi.useRealTimers()
|
||||
})
|
||||
|
||||
async function mount(): Promise<void> {
|
||||
await act(async () => {
|
||||
renderer = create(createElement(Probe))
|
||||
})
|
||||
}
|
||||
|
||||
it('re-asks a bounded number of times when the host does not answer, then shows unavailable', async () => {
|
||||
await mount()
|
||||
expect(reads.calls).toBe(1)
|
||||
for (const delay of [1_500, 5_000]) {
|
||||
await act(async () => {
|
||||
await vi.advanceTimersByTimeAsync(delay)
|
||||
})
|
||||
}
|
||||
expect(reads.calls).toBe(3)
|
||||
expect(latest?.state).toEqual({ kind: 'unavailable' })
|
||||
await act(async () => {
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
})
|
||||
expect(reads.calls).toBe(3)
|
||||
})
|
||||
|
||||
it('does not latch a failure: a retry asks again and can succeed', async () => {
|
||||
reads.next = [{ kind: 'refused' }, { kind: 'ready', html: '<p>', revision: 'a'.repeat(32) }]
|
||||
await mount()
|
||||
expect(latest?.state).toEqual({ kind: 'unavailable' })
|
||||
await act(async () => {
|
||||
latest?.retry()
|
||||
})
|
||||
expect(reads.calls).toBe(2)
|
||||
expect(latest?.state).toEqual({ kind: 'ready', html: '<p>', revision: 'a'.repeat(32) })
|
||||
})
|
||||
|
||||
it('stops re-asking after unmount', async () => {
|
||||
await mount()
|
||||
act(() => renderer?.unmount())
|
||||
renderer = null
|
||||
await act(async () => {
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
})
|
||||
expect(reads.calls).toBe(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,87 @@
|
||||
import { useCallback, useEffect, useState } from 'react'
|
||||
import {
|
||||
cachedMobileNativeChatVisual,
|
||||
readMobileNativeChatVisual,
|
||||
type MobileNativeChatVisualSource
|
||||
} from './mobile-native-chat-visual-read'
|
||||
|
||||
export type MobileNativeChatVisualState =
|
||||
| { kind: 'loading' }
|
||||
| { kind: 'ready'; html: string; revision: string }
|
||||
| { kind: 'unavailable' }
|
||||
|
||||
/** Automatic re-asks after a read that got no answer; the reader's tap starts a fresh round. */
|
||||
const AUTO_RETRY_DELAYS_MS = [1_500, 5_000] as const
|
||||
|
||||
function initialState(
|
||||
source: MobileNativeChatVisualSource,
|
||||
file: string
|
||||
): MobileNativeChatVisualState {
|
||||
const cached = cachedMobileNativeChatVisual(source, file)
|
||||
return cached ? { kind: 'ready', ...cached } : { kind: 'loading' }
|
||||
}
|
||||
|
||||
/**
|
||||
* One visual's content: painted from the phone's cache when it has one, revalidated against the
|
||||
* owning host on every mount, and retried a bounded number of times when the host does not answer.
|
||||
* Nothing latches: `retry` (or a new client after a reconnect) asks again.
|
||||
*/
|
||||
export function useMobileNativeChatVisual(
|
||||
source: MobileNativeChatVisualSource,
|
||||
file: string
|
||||
): { state: MobileNativeChatVisualState; retry: () => void } {
|
||||
const [state, setState] = useState(() => initialState(source, file))
|
||||
// `round` is a reader's retry; `retriesUsed` counts automatic re-asks within it.
|
||||
const [request, setRequest] = useState({ round: 0, retriesUsed: 0 })
|
||||
const [retryInMs, setRetryInMs] = useState<number | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
let disposed = false
|
||||
void readMobileNativeChatVisual(source, file).then((read) => {
|
||||
if (disposed) {
|
||||
return
|
||||
}
|
||||
if (read.kind === 'ready') {
|
||||
// Same revision keeps the same state object, so the frame does not reload its document.
|
||||
setState((current) =>
|
||||
current.kind === 'ready' && current.revision === read.revision
|
||||
? current
|
||||
: { kind: 'ready', html: read.html, revision: read.revision }
|
||||
)
|
||||
return
|
||||
}
|
||||
const delay =
|
||||
read.kind === 'unreachable' ? AUTO_RETRY_DELAYS_MS[request.retriesUsed] : undefined
|
||||
if (delay !== undefined) {
|
||||
setRetryInMs(delay)
|
||||
return
|
||||
}
|
||||
// A host that does not answer leaves a visual on screen; one that refuses it takes it down.
|
||||
setState((current) =>
|
||||
current.kind === 'ready' && read.kind === 'unreachable' ? current : { kind: 'unavailable' }
|
||||
)
|
||||
})
|
||||
return () => {
|
||||
disposed = true
|
||||
}
|
||||
}, [source, file, request])
|
||||
|
||||
useEffect(() => {
|
||||
if (retryInMs === null) {
|
||||
return
|
||||
}
|
||||
const timer = setTimeout(() => {
|
||||
setRetryInMs(null)
|
||||
setRequest((current) => ({ ...current, retriesUsed: current.retriesUsed + 1 }))
|
||||
}, retryInMs)
|
||||
return () => clearTimeout(timer)
|
||||
}, [retryInMs])
|
||||
|
||||
const retry = useCallback(() => {
|
||||
setState((current) => (current.kind === 'unavailable' ? { kind: 'loading' } : current))
|
||||
setRetryInMs(null)
|
||||
setRequest((current) => ({ round: current.round + 1, retriesUsed: 0 }))
|
||||
}, [])
|
||||
|
||||
return { state, retry }
|
||||
}
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
projectStructuredQuestion
|
||||
} from './mobile-structured-agent-prompts'
|
||||
import type { RpcClient } from '../transport/rpc-client'
|
||||
import type { MobileNativeChatVisualSource } from './mobile-native-chat-visual-read'
|
||||
import type { MobileChatPermission } from './mobile-native-chat-permission'
|
||||
import type { MobileChatQuestion } from './mobile-native-chat-question'
|
||||
import type { MobileNativeChatSession } from './use-mobile-native-chat-session'
|
||||
@@ -67,6 +68,8 @@ type StructuredMobileSession = ReturnType<typeof useMobileStructuredAgentOptions
|
||||
cancelPrompt: (prompt?: { itemId: string; expectedRevision: number }) => Promise<boolean>
|
||||
/** The queued-draft cards and their actions, from any host that publishes them. */
|
||||
queued: MobileStructuredQueuedMessageControls
|
||||
/** Where this chat's `::orca-visual` lines read their HTML from; null without a client. */
|
||||
visualSource: MobileNativeChatVisualSource | null
|
||||
}
|
||||
|
||||
export function useMobileStructuredAgentSession(args: {
|
||||
@@ -266,8 +269,14 @@ export function useMobileStructuredAgentSession(args: {
|
||||
]
|
||||
)
|
||||
|
||||
const visualSource = useMemo<MobileNativeChatVisualSource | null>(
|
||||
() => (client && sessionId ? { client, sessionId } : null),
|
||||
[client, sessionId]
|
||||
)
|
||||
|
||||
return {
|
||||
...options,
|
||||
visualSource,
|
||||
session: {
|
||||
messages,
|
||||
status,
|
||||
|
||||
@@ -161,6 +161,10 @@
|
||||
"file": "src/components/AuthFailedBannerActions.web.tsx",
|
||||
"reason": "The auth-failed banner offers Retry, Re-pair and Remove, and on the page only Re-pair works. /pair-scan sits outside the page's route root of app/h (config/scripts/mobile-web-app-route-manifest.mjs:6), so route-handoff.web.ts:207 hands the push to the shell, which opens the native scan screen and returns to the same page document on Back. Retry and Remove are inert here: forceReconnect is null (client-context.web.tsx:57, read through host-client-hooks.ts:88) and removal refuses (page-host-removal-refusal.ts). The banner is reachable on the page because the shell forwards the native client's state (mobile-web-shell/bridge-host.ts:301) and auth-failed is in the wire enum (bridge/bridge-envelope.ts:44). So this one renders Re-pair as native does and one line naming the app for reconnect and removal."
|
||||
},
|
||||
{
|
||||
"file": "src/session/MobileNativeChatVisualFrame.web.tsx",
|
||||
"reason": "The native frame runs a chat visual's scripts in a sandboxed child of a trusted host document inside a react-native-webview, which throws at import in a browser. The page's response-header policy (script-src 'self', frame-src 'none') is inherited by a srcdoc frame, so a visual's inline scripts cannot run here without loosening the page policy. This one renders the visual sealed, as MobileHtmlPreview.web.tsx does: an empty sandbox, no script, a fixed-height frame the visual scrolls inside."
|
||||
},
|
||||
{
|
||||
"file": "src/components/MobileHtmlPreview.web.tsx",
|
||||
"reason": "The native preview renders an agent-produced HTML artifact inside a sandboxed WebView with navigation locked to the initial inline document, and react-native-webview throws at import in a browser for the reason above. This one renders the artifact in a sandboxed iframe with no allow-scripts and no allow-same-origin, keeping the Preview/Source toggle. srcdoc rather than a blob: URL and no CSP change at all: a srcdoc frame has no URL for frame-src to match and inherits its embedder's policy instead, so it is admitted under the shipped frame-src 'none' on Chromium and WebKit alike, while a blob: frame is refused by frame-src and refused again in WebKit by the frame-ancestors 'none' it inherits. The inherited policy is also what seals it -- script-src 'self' refuses the artifact's inline script, img-src bounds its images, font-src 'none' its fonts -- and allow-top-navigation-by-user-activation is the one capability granted, so a tapped link becomes a top-frame navigation the shell opens externally (ruling 29) while a meta refresh, a form submit, target=_blank and any script-initiated navigation produce none. That token is granted only when the shell says it can open such a navigation: without the externalNavigation grant this file renders the artifact with its links as text and seals the frame with an empty sandbox instead (C8.1, ruling 37.2)."
|
||||
|
||||
@@ -170,6 +170,7 @@ describe('mobile RPC allowlist', () => {
|
||||
'agentSession.subscribe',
|
||||
'agentSession.unsubscribe',
|
||||
'agentSession.subscribeStatus',
|
||||
'agentSession.readVisual',
|
||||
'agentSession.hold',
|
||||
'agentSession.release'
|
||||
])
|
||||
|
||||
@@ -23,6 +23,8 @@ export const MOBILE_AGENT_SESSION_RPC_METHODS = [
|
||||
'agentSession.unsubscribe',
|
||||
// Every session's status on one stream: the phone's chat reads the host's "Stopping…" from it.
|
||||
'agentSession.subscribeStatus',
|
||||
// One visual from the chat's own visuals folder, for the transcript's `::orca-visual` lines.
|
||||
'agentSession.readVisual',
|
||||
// No-ops on a current host; kept until MIN_COMPATIBLE_RUNTIME_CLIENT_VERSION passes the
|
||||
// mobile builds that still call them.
|
||||
'agentSession.hold',
|
||||
|
||||
Reference in New Issue
Block a user