From a91ca8b19e6b48b88f49c9bcf7e5941aedd2a1df Mon Sep 17 00:00:00 2001 From: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com> Date: Mon, 21 Sep 2026 06:42:30 -0400 Subject: [PATCH] feat(mobile): dictation on the OTA page over native audio verbs (Phase C, C7.10 PR D) (#21905) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(mobile): serve dictation capture over four native audio verbs The page owns dictation's state machine and speaks `speech.dictation.*` to the desktop, where transcription runs; the microphone is the shell's. So the shell gains `native.audio.start|read|stop` and `native.wakelock.set` — four rows, four grants — and rings what the microphone produces at the page's own pending-audio budget rather than pushing bytes the page would hand straight back. `native_audio_not_capturing` joins the refusal vocabulary: a read for a capture this session does not have is the one refusal the page must tell from a device that failed. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * refactor(mobile): take dictation capture through one seam on both hosts `use-mobile-dictation.ts` held the microphone and the wake tag directly, so the page had a hook whose every device call was a stub answering denied. The five calls move behind `src/platform/dictation-capture.ts`: natively the same calls in the same order, on the page the shell's four verbs, with the drain raising the events the engine emits. The tag bookkeeping stays where it was and stops importing `expo-keep-awake`: two calls come in through the seam and the pools, the queue, the timeouts and the retries are the same on either host. The chunk sender is untouched. A chunk carries raw PCM because that is what the budget counts and what `speech.dictation.chunk` is built from, so the page pays one decode of 32 KB a second rather than the flow carrying two shapes. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * perf(mobile): drain the shell's audio ring on a 500 ms batch One `speech.dictation.chunk` per native microphone event is 31.25 forwarded requests a second, and each holds one of the bridge's 64 in-flight slots for a whole desktop round trip. Measured over ten seconds against a two-second link: 63 in flight at the peak and one slot left for the rest of the page. Drained every 500 ms instead: 5 in flight, 60 slots free, the same 42 KiB/s, and 38 frames out and 34 back for the whole session. The frame cap was never the bound — half a second of PCM is 3.3% of one. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): derive dictation's grant rows from the route closure Which routes need the four audio verbs is a census, not a hand list: the rule reads each registered page route's own closure and holds its `grants` to what that closure reaches. Vacuous on today's route list — the session route is the only closure carrying the seam and is not registered yet — so a control runs the same rule against the session module and names all four as missing. The closure also records what the seam took off the page: `@orca/expo-two-way- audio` and `expo-keep-awake` are gone from it entirely, and removing the web file puts four of the vendored stub's modules back. The mic control's render case found a real one. A start the shell refused outright left the button on "Starting voice dictation" with no way back, which is every tap on a route without the grants. It reports the refusal and returns to idle. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): keep the mic control's render case inside the tests typecheck A `let` the renderer assigns inside a callback narrows to `never` afterwards, and the mocked `Pressable` took `children` as `unknown`. Both are type-level only, and the ratchet is the gate that notices: a test outside `tsc` can pin a shape that stopped existing. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): hand over the audio still in the ring before stopping the shell `end` cancelled the drain and stopped the capture without a last read, so up to one drain interval of the utterance's tail — 16,000 bytes, the 400 ms a user is still speaking as they lift the button — was discarded on every stop. The reviewer's probe spoke 12,288 bytes in the last 400 ms and the page delivered none of them. Natively that audio is already in the hook's hands, so this was a page-only loss of the end of every sentence. `end` is now asynchronous: it cancels the timer, waits for any read in flight, reads once more, and only then stops the shell — stopping first takes the capture away and the read after it is refused. `stop()` awaits it before it stops accepting chunks and before it takes the pending set, or the tail would be dropped one line later and `finish` could overtake the last send. A release still skips the last read: the screen is going away and there is nobody left to hand the tail to. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): release the page session's wake tag when the session ends The wake-lock server held its tags per instance and a new one was built per page session with nothing ever disposing it, so a tag a session took was never given back and the screen stayed awake for the app's lifetime. The page is a document that can navigate, fault or be swiped away mid-dictation, so nothing else was ever going to call deactivate. Its own docstring claimed the opposite. It now answers `{ serve, dispose }` and is disposed with the session exactly as the microphone and the staged media handles are. `dispose` drops only what is still held, so a tag the page already gave back is not deactivated twice. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * refactor(mobile): drop the capability flag no screen reads `canCaptureAudio` answered four grants to nobody and had no test. The fence that actually holds is the per-verb `ungranted` check every member already makes before a frame is sent, and the mic control's render case pins what a screen does with it. The surface's member list is pinned instead, so the next flag with nothing behind it has to be added there on purpose. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): re-measure the closure the dictation census records, and make its controls real Three corrections to the census, all of them about the census lying rather than the product being wrong. The recorded number was four modules; measured on this head it is eight — five from `@orca/expo-two-way-audio` and three from `expo-keep-awake` — for a net +7 once the local file that left is counted. The absolute closure counts are provenance in the docstring and are not asserted, because every merge of main moves them. The absence now has a precondition: both package names are resolved from the install, so a substring matching nothing fails as a typo. The case named "reads the census file" read no file. It reads the shell's own verb table through `import()`, behind the closure guard, so a grant the shell has no row for reds instead of agreeing with itself. And the grant control re-implemented the rule's filter inline. Both the rule and the control drive one function now, over the entry C7.7 would write if it copied its neighbours' grants. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): serialise the shell's audio starts and stops Two starts racing the OS permission prompt — a page reloaded while it is up, which is the case the replacement rule exists for — both reached `listen()`, and the second overwrote the first's handlers without removing them. The engine went on calling into a capture nobody could read, and `dispose` freed one of the two. A stop that overlapped a start found nothing to end and the start opened a microphone behind it. Starts and stops now run one at a time in the order the page asked for them, and a start that comes back after the session ended opens nothing. Reads stay off the queue: they must not wait behind an opening capture, and a read with no capture is already a refusal. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): end a capture on the same two interruptions on both hosts The page treated any interruption as the capture being taken away, while the native seam has always gated on `began` and `blocked`. So an `ended` on its own — the OS handing the session back after a notification chime — cancelled a live dictation on the page and did nothing natively. The rule is now one predicate beside the vocabulary it belongs to, read by all three places that decide it: the shell, which stops filling its ring; the native seam, which raises it off `onAudioInterruption`; and the page, which raises it off a read reply. `recording` still ends the page's capture whatever the kind, because a capture the shell no longer has is gone however it went. The native half had no test of its own, which is why the drift was invisible. It has one now: the five calls it makes, the chunk it hands over, the wake tag, and which interruptions end it. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * refactor(mobile): type the chunk sender as what its callers pass The sender's parameter was the native `MicrophoneDataEvent` though both callers hand it a `DictationCaptureChunk`. Structurally the event is the wider type, so it accepted either and read `droppedBytes` off neither — a page whose audio the shell's ring had dropped would have sent it as though nothing were missing, and nothing would have failed to compile. Typed as the chunk, with a compile fence beside the seam holding both directions: a chunk is accepted, an event is refused, and a raw buffer is not a chunk. The seam normalises the bytes, so the widening the sender did has no caller left. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): re-record the session closure with dictation's page modules on it The capture seam moves this count down rather than up. Measured at 2d697a4012 and at this head: modules 4,324 to 4,319 and local modules 974 to 977. Three local modules join — the page's capture seam, its contract and the audio verb shapes — and eight vendored ones leave, because the seam is what stops the page importing a microphone it does not have: five of `@orca/expo-two-way-audio` and three of `expo-keep-awake`, replaced by four verbs the shell answers. Named beside the sentences already there, with the counterfactual that puts the eight back pointing at the census that runs it. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): re-record the session closure over C2.9's two bridge modules C2.9 moved this pin to 4,326 by putting the page-route grants and the manifest grammar behind `bridge-envelope.ts`, which every page closure reads. Dictation's capture still moves it down from wherever it lands: measured at 5d13a70ea3 and at this head, modules 4,326 to 4,321 and local modules 976 to 979. Three local modules join — the page's capture seam, its contract and the audio verb shapes — and eight vendored ones leave, five of `@orca/expo-two-way-audio` and three of `expo-keep-awake`, replaced by four verbs the shell answers. Recorded beside C2.9's sentences rather than in place of them. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * perf(config): build each route closure once in the dictation census Every `mobileWebAppRouteClosure` call is a full esbuild metafile build, and the cases here ask about six route modules across nine of them — fifteen builds. The CPU that cost tipped two timing-sensitive neighbours in this shard over: a benchmark whose child has 100 ms to write a pid file, and a census globbing a scratch tree another test was removing. Neither is reached by this file and both are fragile without it; the added load was the difference. With the census excluded the shard was green, with it three runs of three were red. Memoised per route module, so six builds. The shard still fails intermittently on this machine for its own reasons, but not because of this file. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(config): gate the dictation census's verb-table case on the mobile install `import()` defers when a module loads, not what loading costs. Vite transforms the file at that moment and resolves the nearest `tsconfig.json` for it, which is `mobile/tsconfig.json`, which extends `expo/tsconfig.base.json` — absent on the root-only shard, so the transform threw `TSConfckParseError` and reddened `test / tests node 24 1/8`. The comment claiming the dynamic import avoided that was wrong. Gated on `mobileWebAppDependenciesPresent()`, the same guard the closure cases use, which is the only thing that keeps a mobile module off that shard. Reproduced both ways in this tree by moving `mobile/node_modules` aside: before, 1 failed with that error; after, 3 passed and 6 skipped; with the install back, 9 passed. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): make the page's capture end idempotent The hook's interruption handler is `() => void cancel()`, and `cancel` reaches `capture.end()` synchronously through `closeDictationAudio`. So the last read in `end` could raise an interruption that called straight back into `end`, whose own last read was refused for the same reason the first was — the shell has no capture — and the recursion issued bridge reads until the page ran out of memory. The pin crashed the test worker with `JavaScript heap out of memory` before the fix. A second `end` returns the first one's promise, assigned before anything can await so a handler re-entering from inside the read finds it set. `begin` clears it, because the seam is memoised per client and the next dictation on the same screen has to be able to drain — pinned by a case that ends, starts again and reads the new audio. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): give back a wake tag whose activation landed after dispose `held.add(tag)` ran after the device call, so a `dispose()` falling between the activate and its reply walked an empty set and the tag was recorded afterwards. Nothing walks that set again, so the screen stayed awake for the app's lifetime — and the page is a document that can be swiped away mid-dictation, which is exactly when that window is open. A tag that lands after the session ended is deactivated on the spot and reported to the page as not held. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): swallow the native audio session's shutdown failures The contract says `end` never rejects, and the native one did: it is `async`, so a throwing JSI binding rejects rather than throws. Every caller reaches it as `void capture.end()` inside a synchronous `try`, which cannot see a rejection — so a device that would not stop recording left an unhandled rejection, and the cleanup the `try` was written to protect was never what was at risk. `release` was worse in kind: it runs bare in the unmount path, so a throwing `tearDown` took the wake tag's release and the desktop's cancel with it. Both log and continue. Pinned behaviourally against an engine that refuses to stop and a tear-down that throws; the source case that claimed the hook's `try` was the guard now says where the guard actually is. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): pin the order stop() hands the tail over in The tail fix depends on `await capture.end()` running before chunks stop being accepted, and nothing held that: the source check only asserts `end()` precedes `Promise.allSettled`, which both orders satisfy, so reversing the two lines left the whole mobile suite green while the page silently dropped the end of every sentence. Driven against a capture whose `end()` delivers a chunk — what the page's seam does and the device's never does, which is why only this case can tell the orders apart. It asserts the tail reaches the desktop as `speech.dictation.chunk`, with those bytes, before `finish`; reversed, no chunk is sent at all. The mocked seam is one object for the module's life, because the hook keys its teardown effect on the capture's identity: a seam returning a fresh object per render cancels the dictation on every render. Both real seams are stable. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): keep a wake tag recorded until the device really drops it `held` was cleared whether or not `deactivate` succeeded, so a tag the device refused was forgotten. The page's owner queues exactly that failure for a retry (`pendingCleanupTags`), and the retry arrives here as another `active: false` — which a shell that had already forgotten the tag answered without calling anything, leaving the native tag on for the life of the app. The boundary: the set means "the device still has this tag", not "the page asked for it". That keeps the reason the set exists — never call `deactivateKeepAwake` for a tag this shell never took, since its failure would read to the page as a wake lock it could not drop — while letting every retry for a live tag through. Always reaching the device regardless of the set would have traded the second property for the first. Pinned with a device that refuses once and then accepts, and with a dispose whose deactivation is refused. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): tear the audio device down for a start that lost the race A start that came back after `dispose()` returned without ending the engine, and the local `end()` is a no-op with no capture, so nothing tore down the session `initialize()` had just brought up. Nobody else would: the dispose had already run and no capture was ever recorded. The device's audio session stayed up for the life of the app. It ends the engine on that path now. Pinned on both places the race can be lost — inside the permission prompt and inside the open itself — each asserting exactly one teardown and no live listeners. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): drive the stop-tail pin through the typed fake client The hand-rolled client fake needed three type assertions to stand in for an `RpcClient`, which the changed-code quality gate refuses on new lines. It drives `createFakeRpcClient` instead — a real one — answering each request as the hook makes it, and reads the chunk's base64 by narrowing rather than asserting. Re-confirmed the pin still reds on the reversed order after the rewrite. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): clear the page capture's end when it settles A finished `end` stayed in `ending` until `begin` cleared it, and `begin` is not guaranteed to run between two ends: `open` starts the shell recording, and a start that goes stale after the hook sets `activeIdRef` cleans up through `capture.end()` without ever committing. The second end answered from the first one's settled promise and never issued `native.audio.stop`, leaving the shell holding a live microphone. Cleared on settle instead, and only for its own flight. The re-entrancy the latch was really for happens while the promise is still pending, so guarding the flight is enough. `release` keeps its own flag rather than a settled `ending`, which now clears itself and would unlatch it. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): serialise the shell's wake tags and record them before compensating Two ways the device could end up holding a tag nothing would ever give back. A release that arrived while its own activate was still in flight read `held` before the activate had recorded anything, found nothing, deactivated nothing and reported the tag off; then the activate landed and the device stayed on. `held` is read and written across an await, so operations are now chained per tag. Per tag rather than per server, so one hanging device call cannot hold up another dictation. And on the late-tag path, a compensating deactivate the device refused was swallowed while `{ active: false }` was returned: the device still held the lock, `held` lacked the tag, `dispose` had already walked the set, and the page believed an activate had succeeded. The tag is now recorded as soon as `activate` resolves, deleted only once the device has really dropped it, and a refusal rejects so the caller's retry path runs. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): let dispose re-read the tag set once its turn comes Queueing dispose behind each tag's own operations introduced a call the module says it does not make: a release already in flight can give the tag back before dispose runs, and deactivating an unheld tag is a native call whose failure would read to the page as a lock it could not drop. Re-reads the set when the queued action runs rather than trusting what it held when dispose was called. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): pin the dictation census to the routes C4.4 registered Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb --- ...web-app-session-dictation-capture.test.mjs | 260 ++++++++ ...-web-app-session-terminal-closure.test.mjs | 19 +- .../src/ExpoTwoWayAudioModule.web.ts | 6 +- .../src/hooks/mobile-dictation-audio-chunk.ts | 8 +- .../mobile-dictation-foreground-keep-awake.ts | 20 +- .../hooks/mobile-dictation-keep-awake.test.ts | 68 +- .../src/hooks/mobile-dictation-keep-awake.ts | 52 +- .../hooks/mobile-dictation-stop-tail.test.tsx | 193 ++++++ .../hooks/use-mobile-dictation-source.test.ts | 88 ++- mobile/src/hooks/use-mobile-dictation.ts | 100 +-- .../MobileWebShellScreen.test.tsx | 12 + .../mobile-web-shell/bridge-host-errors.ts | 6 +- .../mobile-web-shell/bridge-host-init.test.ts | 6 +- .../bridge/bridge-audio-verbs.test.ts | 622 ++++++++++++++++++ .../bridge/bridge-audio-verbs.ts | 162 +++++ .../bridge/bridge-native-verbs.test.ts | 6 +- .../bridge/bridge-native-verbs.ts | 35 +- .../bridge/bridge-port-pair-test-harness.ts | 8 + .../bridge/use-native-verbs.test.tsx | 26 + .../bridge/use-native-verbs.ts | 27 +- .../page-route-policy.test.ts | 6 +- .../dictation-capture-bridge-budget.test.tsx | 277 ++++++++ .../dictation-capture-compile-fence.ts | 35 + .../platform/dictation-capture-contract.ts | 109 +++ mobile/src/platform/dictation-capture.test.ts | 152 +++++ mobile/src/platform/dictation-capture.ts | 75 +++ .../platform/dictation-capture.web.test.tsx | 515 +++++++++++++++ mobile/src/platform/dictation-capture.web.ts | 250 +++++++ mobile/src/platform/native-audio-device.ts | 72 ++ mobile/src/platform/native-audio.ts | 247 +++++++ mobile/src/platform/native-wakelock.test.ts | 140 ++++ mobile/src/platform/native-wakelock.ts | 120 ++++ .../platform/use-native-device-verbs.test.tsx | 58 +- .../src/platform/use-native-device-verbs.ts | 34 +- .../mobile-dictation-mic-control.web.test.tsx | 236 +++++++ mobile/web-entry/web-overrides.json | 6 +- 36 files changed, 3899 insertions(+), 157 deletions(-) create mode 100644 config/scripts/mobile-web-app-session-dictation-capture.test.mjs create mode 100644 mobile/src/hooks/mobile-dictation-stop-tail.test.tsx create mode 100644 mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.test.ts create mode 100644 mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.ts create mode 100644 mobile/src/platform/dictation-capture-bridge-budget.test.tsx create mode 100644 mobile/src/platform/dictation-capture-compile-fence.ts create mode 100644 mobile/src/platform/dictation-capture-contract.ts create mode 100644 mobile/src/platform/dictation-capture.test.ts create mode 100644 mobile/src/platform/dictation-capture.ts create mode 100644 mobile/src/platform/dictation-capture.web.test.tsx create mode 100644 mobile/src/platform/dictation-capture.web.ts create mode 100644 mobile/src/platform/native-audio-device.ts create mode 100644 mobile/src/platform/native-audio.ts create mode 100644 mobile/src/platform/native-wakelock.test.ts create mode 100644 mobile/src/platform/native-wakelock.ts create mode 100644 mobile/src/session/mobile-dictation-mic-control.web.test.tsx diff --git a/config/scripts/mobile-web-app-session-dictation-capture.test.mjs b/config/scripts/mobile-web-app-session-dictation-capture.test.mjs new file mode 100644 index 00000000000..cd6fb84870f --- /dev/null +++ b/config/scripts/mobile-web-app-session-dictation-capture.test.mjs @@ -0,0 +1,260 @@ +/** + * Which route closures reach dictation's capture seam, and therefore which routes must be granted + * the four audio verbs. + * + * A census rather than a hand list, because a grant row written by hand is a row that stops + * agreeing with the closure the moment a screen moves: the rule below reads what each registered + * page route actually reaches and holds its `grants` to it. Vacuous today — the session route is + * the only closure that reaches the seam and `MOBILE_WEB_PAGE_ROUTES` does not carry it yet (C7.7 + * registers it) — so the control beside it applies the same rule to the session route module and + * shows the rule failing without the four names. + * + * The closure also says what the seam took out of the page. Without its web half the bundler + * resolves the native one, and the vendored `@orca/expo-two-way-audio` web stub lands in the + * closure along with `expo-keep-awake` — which is what dictation on the page used to be: a module + * answering denied microphone permission, and a wake lock that did nothing. + * + * Measured on this tree by moving `dictation-capture.web.ts` aside and walking the closure again: + * `modules` 4,319 to 4,326 and `local` 977 to 976. Eight vendored modules re-enter — five from + * `@orca/expo-two-way-audio` and three from `expo-keep-awake` — less the one local file that left, + * which is the +7. The eight is the number below; the absolute counts are provenance and are not + * asserted, because every merge of main moves them and a census that pinned them would fail for + * reasons that are nobody's. + * + * So "absent" here is a fact about the seam and not about the census failing to look, and the + * precondition is checked rather than assumed: both package names are resolved from the install, so + * a substring that matches nothing fails as a typo rather than passing as an absence. + */ +import { createRequire } from 'node:module' +import { fileURLToPath } from 'node:url' +import { describe, expect, it } from 'vitest' +import { mobileWebAppRouteClosure } from './build-mobile-web-app-bundle.mjs' +import { mobileWebAppDependenciesPresent } from './mobile-web-app-bundle-dependencies.mjs' +import { MOBILE_WEB_PAGE_ROUTES } from './mobile-web-page-routes.mjs' +import { MobileWebBundleRouteSchema } from '../../src/shared/mobile-web-bundle/manifest-contract.ts' + +/** The mobile install: the closure builds need it, and so does anything that imports a mobile + * module, because transforming one resolves `mobile/tsconfig.json` and its Expo base. */ +const bundles = mobileWebAppDependenciesPresent() +const describeClosure = bundles ? describe : describe.skip + +/** The seam, as the web build resolves it: `.web.ts` wins under the builder's resolveExtensions. */ +const SEAM = 'src/platform/dictation-capture.web.ts' + +/** The native half, which must resolve out of a page closure rather than sit in it unused. */ +const NATIVE_SEAM = 'src/platform/dictation-capture.ts' + +/** Every verb the seam calls. Named here so the rule below is the census's own answer and not a + * second list to keep true; `bridge-audio-verbs.test.ts` pins them against the verb table. */ +const DICTATION_GRANTS = [ + 'native.audio.start', + 'native.audio.read', + 'native.audio.stop', + 'native.wakelock.set' +] + +/** Native modules the seam exists to keep out: importing either reaches a JSI binding, and their + * web builds are a denied microphone and a no-op wake lock. */ +const NATIVE_AUDIO_MODULES = ['@orca/expo-two-way-audio', 'expo-keep-awake'] + +/** + * How many of their modules re-enter the session closure when the seam's web half is moved aside. + * + * Recorded rather than measured here, because measuring it means walking the closure a second time + * against a mutated tree. Five from `@orca/expo-two-way-audio` (its module, `core`, `events`, + * `hooks` and the index) and three from `expo-keep-awake`. The docstring above carries the run. + */ +const NATIVE_AUDIO_MODULES_BEHIND_THE_SEAM = 8 + +const SESSION_PATHNAME = '/h/[hostId]/session/[worktreeId]' +const SESSION = 'app/h/[hostId]/session/[worktreeId].tsx' + +/** + * One closure per route module, built once. + * + * Every call to `mobileWebAppRouteClosure` is a full esbuild metafile build of the route, and the + * cases below ask about six modules across nine of them. Unmemoised this file did fifteen builds and + * its CPU tipped two timing-sensitive neighbours in this shard over — a benchmark whose child had + * 100 ms to write a pid file, and a census globbing a scratch tree another test was removing. Both + * are fragile without this file and neither is reached by it; the load was the difference. + */ +const closures = new Map() + +function closureOf(routeModule) { + const built = closures.get(routeModule) ?? mobileWebAppRouteClosure(routeModule) + closures.set(routeModule, built) + return built +} + +/** Resolved from `mobile/`, which is the tree the bundler resolves the closure out of: this suite + * runs at the repo root, where neither package is installed. */ +function resolveFromMobile(specifier) { + return createRequire(new URL('../../mobile/package.json', import.meta.url)).resolve(specifier) +} + +/** The route module a registered pathname is served from, the way expo-router files are named. */ +function routeModule(pathname) { + const withoutRoot = pathname.replace(/^\//, '') + const last = withoutRoot.split('/').at(-1) + return last === '[hostId]' ? `app/${withoutRoot}/index.tsx` : `app/${withoutRoot}.tsx` +} + +/** The grants a closure needs of the seam: all four, or none. A route granted three would record + * with the screen free to lock, and a lock mid-processing suspends the app and loses the + * transcript. */ +function dictationGrantsNeeded(closure) { + return closure.local.includes(SEAM) ? DICTATION_GRANTS : [] +} + +/** + * The rule, as one function both the check and its control drive. + * + * Every grant a route's own closure needs and its entry does not name, as ` needs + * `. One implementation, because a control that re-implemented the filter would prove the + * control works and say nothing about the rule. + */ +async function grantsMissingForRoutes(routes) { + const missing = [] + for (const route of routes) { + const closure = await closureOf(routeModule(route.pathname)) + for (const grant of dictationGrantsNeeded(closure)) { + if (!route.grants.includes(grant)) { + missing.push(`${route.pathname} needs ${grant}`) + } + } + } + return missing +} + +describeClosure( + 'the routes that reach dictation capture', + () => { + it('holds every registered page route to the grants its own closure needs', async () => { + expect(await grantsMissingForRoutes(MOBILE_WEB_PAGE_ROUTES)).toEqual([]) + }) + + it('finds the seam in exactly one closure, which is the session route', async () => { + const reaching = [] + for (const route of MOBILE_WEB_PAGE_ROUTES) { + const closure = await closureOf(routeModule(route.pathname)) + if (closure.local.includes(SEAM)) { + reaching.push(route.pathname) + } + } + // None today: dictation lives on the session screen, and that route is not registered yet. + // Which is why the rule above passes without a grant row moving, and why the control below + // is what proves the rule can fail at all. + expect(reaching).toEqual([]) + const session = await closureOf(SESSION) + expect(session.local).toContain(SEAM) + }) + + it('reds the same rule when the session route is registered without them', async () => { + // The control for the rule above, which is vacuous until C7.7 registers this route: the same + // loop, driven over the entry C7.7 would write if it copied its neighbours' grants. + expect( + await grantsMissingForRoutes([ + { pathname: SESSION_PATHNAME, grants: ['navigate', 'storage'] } + ]) + ).toEqual(DICTATION_GRANTS.map((grant) => `${SESSION_PATHNAME} needs ${grant}`)) + // And with all four named it passes, so the rule is satisfiable and not a wall. + expect( + await grantsMissingForRoutes([ + { pathname: SESSION_PATHNAME, grants: ['navigate', 'storage', ...DICTATION_GRANTS] } + ]) + ).toEqual([]) + // Every one of the four is a name a manifest route may carry, which is the ruling-6a trap: + // `native.audio.readChunk` is not a route that degrades to native, it is a bundle the phone + // refuses entire. + expect( + MobileWebBundleRouteSchema.safeParse({ + pathname: SESSION_PATHNAME, + grants: DICTATION_GRANTS + }).success + ).toBe(true) + }) + + it('carries the seam and not the native audio chain it stands in for', async () => { + const closure = await closureOf(SESSION) + expect(closure.local).toContain(SEAM) + expect(closure.local).not.toContain(NATIVE_SEAM) + for (const absent of NATIVE_AUDIO_MODULES) { + // The precondition for reading an absence: the package is installed, so the substring below + // would match if the closure carried it. Without this the case passes on a typo. + expect(() => resolveFromMobile(`${absent}/package.json`), absent).not.toThrow() + expect( + closure.modules.filter((module) => module.includes(`/${absent}/`)), + absent + ).toEqual([]) + } + // The hook above the seam is still in the closure, so the absences above are the seam's work + // and not dictation having left the page. + expect(closure.local).toContain('src/hooks/use-mobile-dictation.ts') + expect(closure.local).toContain('src/hooks/mobile-dictation-keep-awake.ts') + }) + + it('is big enough that finding nothing would mean something', async () => { + const closure = await closureOf(SESSION) + // The largest route of the series; a closure that collapsed would pass every rule above by + // containing nothing to judge. + expect(closure.local.length).toBeGreaterThan(900) + }) + }, + 240_000 +) + +describe('the census rule itself', () => { + it('names a route module for every registered pathname', () => { + expect(MOBILE_WEB_PAGE_ROUTES.map((route) => routeModule(route.pathname))).toEqual([ + 'app/h/[hostId]/index.tsx', + 'app/h/[hostId]/agent-history/[worktreeId].tsx', + 'app/h/[hostId]/tasks.tsx', + 'app/h/[hostId]/files/[worktreeId].tsx', + 'app/h/[hostId]/files/preview/[worktreeId].tsx', + 'app/h/[hostId]/source-control/[worktreeId].tsx', + 'app/h/[hostId]/review/[worktreeId].tsx' + ]) + }) + + it('asks for all four grants or none, never a subset', () => { + expect(dictationGrantsNeeded({ local: [SEAM] })).toEqual(DICTATION_GRANTS) + expect(dictationGrantsNeeded({ local: ['src/platform/media-picker.web.ts'] })).toEqual([]) + }) + + it('records what the seam keeps out, in the number that was measured', () => { + expect(NATIVE_AUDIO_MODULES_BEHIND_THE_SEAM).toBe(8) + expect(NATIVE_AUDIO_MODULES).toHaveLength(2) + }) + + /** + * Gated on the mobile install, not merely deferred behind `import()`. + * + * A dynamic import defers *when* the module loads, not what loading costs. Vite transforms the + * file at that moment and resolves the nearest `tsconfig.json` for it, which is + * `mobile/tsconfig.json`, which extends `expo/tsconfig.base.json`. On the root-only shard that + * package is not installed and the transform throws `TSConfckParseError` — reproduced by moving + * `mobile/node_modules` aside and running this file from the repo root, which is what + * `test / tests node 24 1/8` does. So the dependency gate is the only thing that keeps a mobile + * module off that shard, and it is the same gate `describeClosure` above uses. + */ + it.skipIf(!bundles)( + 'names only verbs the shell actually serves, read from its own table', + async () => { + // The failure this guards is the rule agreeing with itself: a list of four names the census + // holds routes to, none of which the shell has a row for. + const { BRIDGE_NATIVE_VERB_NAMES } = + await import('../../mobile/src/mobile-web-shell/bridge/bridge-native-verbs.ts') + expect(new Set(DICTATION_GRANTS).size).toBe(4) + for (const grant of DICTATION_GRANTS) { + expect(BRIDGE_NATIVE_VERB_NAMES, grant).toContain(grant) + expect( + MobileWebBundleRouteSchema.safeParse({ pathname: '/h', grants: [grant] }).success, + grant + ).toBe(true) + } + } + ) +}) + +/** Kept so a reader can find the tree this ran against without a machine path in the file. */ +export const MOBILE_DIR = fileURLToPath(new URL('../../mobile/', import.meta.url)) diff --git a/config/scripts/mobile-web-app-session-terminal-closure.test.mjs b/config/scripts/mobile-web-app-session-terminal-closure.test.mjs index 8012021e52e..45bce378d43 100644 --- a/config/scripts/mobile-web-app-session-terminal-closure.test.mjs +++ b/config/scripts/mobile-web-app-session-terminal-closure.test.mjs @@ -164,8 +164,25 @@ const MERMAID_PACKAGE = 'node_modules/mermaid/' * The module list on the merge, recorded at the base in the docstring above, which is where every * part of it is accounted for: the document's own modules replacing the factory that carried them, * mermaid's three, and the three bridge modules #21908 and C2.9 pin on main. + * + * Then C7.10 item D put dictation's capture on the page, and the list moved down rather than up. + * + * modules 4328 -> 4323 (-5) + * local modules 978 -> 981 (+3) + * + * Three local modules join — `src/platform/dictation-capture.web.ts`, its contract + * `src/platform/dictation-capture-contract.ts`, and the verb shapes in + * `src/mobile-web-shell/bridge/bridge-audio-verbs.ts` — and eight vendored ones leave, because the + * capture seam is what stops the page importing a microphone it does not have. Five are + * `@orca/expo-two-way-audio` (its web module, `core`, `events`, `hooks` and the index) and three + * are `expo-keep-awake`; the page asks the shell for both over `native.audio.start|read|stop` and + * `native.wakelock.set` instead. The native halves of the seam resolve out of this closure + * entirely, which is the -8 + 3. + * + * Measured, not derived: `mobile-web-app-session-dictation-capture.test.mjs` moves the web file + * aside and walks the closure again, which puts those eight back. */ -const SESSION_ROUTE_MODULES = 4328 +const SESSION_ROUTE_MODULES = 4323 const artifactModules = (inputs) => inputs.filter((input) => input.includes(MERMAID_PAGE_ENGINE)) const packageModules = (inputs) => inputs.filter((input) => input.includes(MERMAID_PACKAGE)) diff --git a/mobile/packages/expo-two-way-audio/src/ExpoTwoWayAudioModule.web.ts b/mobile/packages/expo-two-way-audio/src/ExpoTwoWayAudioModule.web.ts index 5684579fd4c..c4c9c58ceb4 100644 --- a/mobile/packages/expo-two-way-audio/src/ExpoTwoWayAudioModule.web.ts +++ b/mobile/packages/expo-two-way-audio/src/ExpoTwoWayAudioModule.web.ts @@ -33,8 +33,10 @@ const deniedMicrophonePermission: PermissionResponse = { const noop = () => undefined const ExpoTwoWayAudioModule: ExpoTwoWayAudioWebModule = { - // Why: the mobile app can be run on web for QA, but dictation depends on - // native audio engines that are only available in the iOS/Android builds. + // Why: this is what a browser outside the Orca shell can honestly say. Dictation on the page no + // longer comes through here — `src/platform/dictation-capture.web.ts` asks the shell for the + // microphone over `native.audio.start|read|stop`, so the only importer of this package is the + // native half of that seam. What is left is the QA web build, which has no shell to ask. initialize: async () => false, playPCMData: noop, bypassVoiceProcessing: noop, diff --git a/mobile/src/hooks/mobile-dictation-audio-chunk.ts b/mobile/src/hooks/mobile-dictation-audio-chunk.ts index bb094f04807..e4ffe24f6e4 100644 --- a/mobile/src/hooks/mobile-dictation-audio-chunk.ts +++ b/mobile/src/hooks/mobile-dictation-audio-chunk.ts @@ -4,7 +4,7 @@ import { } from './mobile-dictation-pending-audio-budget' import { bytesToBase64 } from './mobile-dictation-session-state' import { dictationAudioChunkSend } from '../dictation/mobile-dictation-operations' -import type { MicrophoneDataEvent } from '@orca/expo-two-way-audio' +import type { DictationCaptureChunk } from '../platform/dictation-capture-contract' import type { MobileDictationPendingAudioBudget } from './mobile-dictation-pending-audio-budget' import type { RpcClient } from '../transport/rpc-client' @@ -18,11 +18,11 @@ type MobileDictationAudioChunkQueue = { export function enqueueMobileDictationAudioChunk( client: RpcClient, dictationId: string, - event: MicrophoneDataEvent, + chunk: DictationCaptureChunk, queue: MobileDictationAudioChunkQueue ): void { - const raw = event.data - const bytes = raw instanceof Uint8Array ? raw : new Uint8Array(raw) + // The seam normalises what the engine handed over, so there is nothing to widen here. + const bytes = chunk.data const byteLength = bytes.byteLength if (!queue.pendingAudioBudget.tryReserve(byteLength)) { queue.failActiveDictation( diff --git a/mobile/src/hooks/mobile-dictation-foreground-keep-awake.ts b/mobile/src/hooks/mobile-dictation-foreground-keep-awake.ts index d5962d9d11e..e5571e0379d 100644 --- a/mobile/src/hooks/mobile-dictation-foreground-keep-awake.ts +++ b/mobile/src/hooks/mobile-dictation-foreground-keep-awake.ts @@ -2,6 +2,7 @@ import { useEffect } from 'react' import { AppState, Platform } from 'react-native' import { drainMobileDictationKeepAwakeCleanup } from './mobile-dictation-keep-awake' import type { RefObject } from 'react' +import type { DictationKeepAwakeDevice } from '../platform/dictation-capture-contract' import type { MobileDictationKeepAwakeOwner } from './mobile-dictation-keep-awake' // A transient Activity gap can fail a foreground refresh; retry briefly while @@ -9,29 +10,36 @@ import type { MobileDictationKeepAwakeOwner } from './mobile-dictation-keep-awak const REACQUIRE_RETRY_DELAYS_MS = [1_000, 5_000] let globalStaleTagDrainInstalled = false +// The drain outlives every owner, so it reads the newest device rather than capturing one: on the +// page that device is built from a bridge client the screen can replace, and a captured one would +// deactivate through a port nothing is listening on. +let latestKeepAwakeDevice: DictationKeepAwakeDevice | null = null // Failed final deactivations must be retried even after every session screen // unmounts, or a stale native tag keeps the screen awake until app restart. // Installed once for the app's lifetime; the drain spares still-wanted tags // and fast-paths to a no-op when nothing is pending. -function installGlobalStaleTagForegroundDrain(): void { +function installGlobalStaleTagForegroundDrain(device: DictationKeepAwakeDevice): void { + latestKeepAwakeDevice = device if (globalStaleTagDrainInstalled) { return } globalStaleTagDrainInstalled = true AppState.addEventListener('change', (state) => { - if (state === 'active') { - void drainMobileDictationKeepAwakeCleanup().catch(() => undefined) + const current = latestKeepAwakeDevice + if (state === 'active' && current !== null) { + void drainMobileDictationKeepAwakeCleanup(current).catch(() => undefined) } }) } export function useMobileDictationForegroundKeepAwake( keepAwakeOwner: MobileDictationKeepAwakeOwner, - activeIdRef: RefObject + activeIdRef: RefObject, + keepAwakeDevice: DictationKeepAwakeDevice ): void { useEffect(() => { - installGlobalStaleTagForegroundDrain() + installGlobalStaleTagForegroundDrain(keepAwakeDevice) // Android keeps FLAG_KEEP_SCREEN_ON on the Activity window, so Activity // recreation silently drops it mid-dictation; refresh on return to // active. iOS re-applies natively on foreground. @@ -66,5 +74,5 @@ export function useMobileDictationForegroundKeepAwake( reacquireRun += 1 sub.remove() } - }, [keepAwakeOwner, activeIdRef]) + }, [keepAwakeOwner, activeIdRef, keepAwakeDevice]) } diff --git a/mobile/src/hooks/mobile-dictation-keep-awake.test.ts b/mobile/src/hooks/mobile-dictation-keep-awake.test.ts index 362f3a0cf68..52e35947903 100644 --- a/mobile/src/hooks/mobile-dictation-keep-awake.test.ts +++ b/mobile/src/hooks/mobile-dictation-keep-awake.test.ts @@ -1,21 +1,19 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' - -const keepAwake = vi.hoisted(() => ({ - activate: vi.fn<(tag: string) => Promise>(), - deactivate: vi.fn<(tag: string) => Promise>() -})) - -vi.mock('expo-keep-awake', () => ({ - activateKeepAwakeAsync: keepAwake.activate, - deactivateKeepAwake: keepAwake.deactivate -})) - import { MOBILE_DICTATION_KEEP_AWAKE_NATIVE_TIMEOUT_MS, MobileDictationKeepAwakeOwner, drainMobileDictationKeepAwakeCleanup } from './mobile-dictation-keep-awake' +/** The two calls the owner makes, which on a device are `expo-keep-awake` and on the page are + * `native.wakelock.set`. Everything under test here is what the owner does around them. */ +const keepAwake = { + activate: vi.fn<(tag: string) => Promise>(), + deactivate: vi.fn<(tag: string) => Promise>() +} + +const device = { activate: keepAwake.activate, deactivate: keepAwake.deactivate } + function deferred(): { promise: Promise resolve: () => void @@ -41,7 +39,7 @@ describe('MobileDictationKeepAwakeOwner', () => { }) it('retries a failed native deactivation after the hook owner is replaced', async () => { - const firstOwner = new MobileDictationKeepAwakeOwner() + const firstOwner = new MobileDictationKeepAwakeOwner(device) await firstOwner.acquire('first') const firstTag = keepAwake.activate.mock.calls[0]?.[0] @@ -50,7 +48,7 @@ describe('MobileDictationKeepAwakeOwner', () => { keepAwake.deactivate.mockRejectedValueOnce(new Error('Activity unavailable')) await expect(firstOwner.release('first')).rejects.toThrow('Activity unavailable') - const replacementOwner = new MobileDictationKeepAwakeOwner() + const replacementOwner = new MobileDictationKeepAwakeOwner(device) await replacementOwner.acquire('second') const secondTag = keepAwake.activate.mock.calls[1]?.[0] expect(secondTag).toContain(':second') @@ -65,7 +63,7 @@ describe('MobileDictationKeepAwakeOwner', () => { it('serializes cancel and restart without letting a stale release deactivate the restart', async () => { const firstActivation = deferred() keepAwake.activate.mockImplementationOnce(() => firstActivation.promise) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) const acquireFirst = owner.acquire('first') const releaseFirst = owner.release('first') @@ -83,7 +81,7 @@ describe('MobileDictationKeepAwakeOwner', () => { it('waits for an in-flight failed release before a replacement owner activates', async () => { const deactivation = deferred() - const firstOwner = new MobileDictationKeepAwakeOwner() + const firstOwner = new MobileDictationKeepAwakeOwner(device) await firstOwner.acquire('first') keepAwake.deactivate.mockImplementationOnce(() => deactivation.promise) @@ -91,7 +89,7 @@ describe('MobileDictationKeepAwakeOwner', () => { await new Promise((resolve) => setTimeout(resolve, 0)) expect(keepAwake.deactivate).toHaveBeenCalledOnce() - const replacementOwner = new MobileDictationKeepAwakeOwner() + const replacementOwner = new MobileDictationKeepAwakeOwner(device) const acquireReplacement = replacementOwner.acquire('replacement') expect(keepAwake.activate).toHaveBeenCalledOnce() @@ -108,7 +106,7 @@ describe('MobileDictationKeepAwakeOwner', () => { }) it('does not fail a fresh acquire when stale-tag cleanup keeps failing', async () => { - const firstOwner = new MobileDictationKeepAwakeOwner() + const firstOwner = new MobileDictationKeepAwakeOwner(device) await firstOwner.acquire('first') // Both the release deactivate and its trailing drain retry fail. @@ -118,7 +116,7 @@ describe('MobileDictationKeepAwakeOwner', () => { await expect(firstOwner.release('first')).rejects.toThrow('Activity unavailable') keepAwake.deactivate.mockRejectedValueOnce(new Error('Activity unavailable')) - const replacementOwner = new MobileDictationKeepAwakeOwner() + const replacementOwner = new MobileDictationKeepAwakeOwner(device) await expect(replacementOwner.acquire('second')).resolves.toBeUndefined() expect(keepAwake.activate).toHaveBeenCalledTimes(2) @@ -133,7 +131,7 @@ describe('MobileDictationKeepAwakeOwner', () => { vi.useFakeTimers() try { keepAwake.activate.mockImplementationOnce(() => new Promise(() => undefined)) - const hungOwner = new MobileDictationKeepAwakeOwner() + const hungOwner = new MobileDictationKeepAwakeOwner(device) const hungAcquire = hungOwner.acquire('hung') // Drain microtasks to quiescence so the timeout timer is registered. await vi.advanceTimersByTimeAsync(0) @@ -143,7 +141,7 @@ describe('MobileDictationKeepAwakeOwner', () => { // The queue must advance, and another owner's drain must spare the // still-wanted maybe-late activation. - const nextOwner = new MobileDictationKeepAwakeOwner() + const nextOwner = new MobileDictationKeepAwakeOwner(device) await nextOwner.acquire('next') expect(keepAwake.deactivate).not.toHaveBeenCalled() expect(keepAwake.activate.mock.calls[1]?.[0]).toContain(':next') @@ -164,7 +162,7 @@ describe('MobileDictationKeepAwakeOwner', () => { try { const lateActivation = deferred() keepAwake.activate.mockImplementationOnce(() => lateActivation.promise) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) const acquire = owner.acquire('late') await vi.advanceTimersByTimeAsync(0) @@ -190,13 +188,13 @@ describe('MobileDictationKeepAwakeOwner', () => { try { const lateActivation = deferred() keepAwake.activate.mockImplementationOnce(() => lateActivation.promise) - const ownerA = new MobileDictationKeepAwakeOwner() + const ownerA = new MobileDictationKeepAwakeOwner(device) const acquireA = ownerA.acquire('wanted') await vi.advanceTimersByTimeAsync(0) await vi.advanceTimersByTimeAsync(MOBILE_DICTATION_KEEP_AWAKE_NATIVE_TIMEOUT_MS) await expect(acquireA).rejects.toThrow('Keep-awake native call timed out') - const ownerB = new MobileDictationKeepAwakeOwner() + const ownerB = new MobileDictationKeepAwakeOwner(device) await ownerB.acquire('other') expect(keepAwake.deactivate).not.toHaveBeenCalled() @@ -218,7 +216,7 @@ describe('MobileDictationKeepAwakeOwner', () => { try { const lateActivation = deferred() keepAwake.activate.mockImplementationOnce(() => lateActivation.promise) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) const acquire = owner.acquire('ended') await vi.advanceTimersByTimeAsync(0) await vi.advanceTimersByTimeAsync(MOBILE_DICTATION_KEEP_AWAKE_NATIVE_TIMEOUT_MS) @@ -236,7 +234,7 @@ describe('MobileDictationKeepAwakeOwner', () => { it('retries a timed-out final deactivation via the foreground drain', async () => { vi.useFakeTimers() try { - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) await owner.acquire('final') const tag = keepAwake.activate.mock.calls[0]?.[0] // The release deactivate times out and its trailing drain retry fails. @@ -249,7 +247,7 @@ describe('MobileDictationKeepAwakeOwner', () => { await expect(release).rejects.toThrow('Keep-awake native call timed out') expect(keepAwake.deactivate).toHaveBeenCalledTimes(2) - await drainMobileDictationKeepAwakeCleanup() + await drainMobileDictationKeepAwakeCleanup(device) expect(keepAwake.deactivate).toHaveBeenCalledTimes(3) expect(keepAwake.deactivate).toHaveBeenLastCalledWith(tag) } finally { @@ -261,7 +259,7 @@ describe('MobileDictationKeepAwakeOwner', () => { vi.useFakeTimers() try { keepAwake.activate.mockImplementationOnce(() => new Promise(() => undefined)) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) const acquire = owner.acquire('orphan') await vi.advanceTimersByTimeAsync(0) await vi.advanceTimersByTimeAsync(MOBILE_DICTATION_KEEP_AWAKE_NATIVE_TIMEOUT_MS) @@ -279,7 +277,7 @@ describe('MobileDictationKeepAwakeOwner', () => { it('recovers on foreground reacquire after a failed initial acquisition', async () => { keepAwake.activate.mockRejectedValueOnce(new Error('Unable to activate keep awake')) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) await expect(owner.acquire('current')).rejects.toThrow('Unable to activate keep awake') expect(keepAwake.deactivate).not.toHaveBeenCalled() @@ -295,7 +293,7 @@ describe('MobileDictationKeepAwakeOwner', () => { }) it('recovers keep-awake on a later reacquire after a failed refresh', async () => { - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) await owner.acquire('current') const tag = keepAwake.activate.mock.calls[0]?.[0] @@ -314,7 +312,7 @@ describe('MobileDictationKeepAwakeOwner', () => { }) it('records new-dictation intent even when previous-tag cleanup fails', async () => { - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) await owner.acquire('first') // Release and its trailing drain both fail; the owner keeps stale intent. @@ -343,7 +341,7 @@ describe('MobileDictationKeepAwakeOwner', () => { try { const first = deferred() keepAwake.activate.mockImplementationOnce(() => first.promise) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) const acquire = owner.acquire('stacked') await vi.advanceTimersByTimeAsync(0) await vi.advanceTimersByTimeAsync(MOBILE_DICTATION_KEEP_AWAKE_NATIVE_TIMEOUT_MS) @@ -376,7 +374,7 @@ describe('MobileDictationKeepAwakeOwner', () => { vi.useFakeTimers() try { keepAwake.activate.mockImplementationOnce(() => new Promise(() => undefined)) - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) const acquire = owner.acquire('maybe') await vi.advanceTimersByTimeAsync(0) await vi.advanceTimersByTimeAsync(MOBILE_DICTATION_KEEP_AWAKE_NATIVE_TIMEOUT_MS) @@ -398,7 +396,7 @@ describe('MobileDictationKeepAwakeOwner', () => { }) it('keeps a live tag out of the orphan pool when a refresh deactivation fails', async () => { - const ownerA = new MobileDictationKeepAwakeOwner() + const ownerA = new MobileDictationKeepAwakeOwner(device) await ownerA.acquire('live') const liveTag = keepAwake.activate.mock.calls[0]?.[0] @@ -409,7 +407,7 @@ describe('MobileDictationKeepAwakeOwner', () => { expect(keepAwake.deactivate.mock.calls.filter(([tag]) => tag === liveTag)).toHaveLength(1) // Another owner's drain must spare the still-wanted live tag. - const ownerB = new MobileDictationKeepAwakeOwner() + const ownerB = new MobileDictationKeepAwakeOwner(device) await ownerB.acquire('other') expect(keepAwake.deactivate.mock.calls.filter(([tag]) => tag === liveTag)).toHaveLength(1) @@ -422,7 +420,7 @@ describe('MobileDictationKeepAwakeOwner', () => { }) it('reacquires by deactivating before activating so Android re-applies the window flag', async () => { - const owner = new MobileDictationKeepAwakeOwner() + const owner = new MobileDictationKeepAwakeOwner(device) await owner.acquire('current') const tag = keepAwake.activate.mock.calls[0]?.[0] diff --git a/mobile/src/hooks/mobile-dictation-keep-awake.ts b/mobile/src/hooks/mobile-dictation-keep-awake.ts index f94e3fb785a..f99578b9ba3 100644 --- a/mobile/src/hooks/mobile-dictation-keep-awake.ts +++ b/mobile/src/hooks/mobile-dictation-keep-awake.ts @@ -1,4 +1,4 @@ -import { activateKeepAwakeAsync, deactivateKeepAwake } from 'expo-keep-awake' +import type { DictationKeepAwakeDevice } from '../platform/dictation-capture-contract' const MOBILE_DICTATION_KEEP_AWAKE_TAG_PREFIX = 'orca-mobile-dictation' @@ -51,8 +51,12 @@ function withNativeCallTimeout(nativeCall: Promise): Promise { }) } -async function activateTrackedTag(tag: string, isStillWanted: () => boolean): Promise { - const nativeActivation = activateKeepAwakeAsync(tag) +async function activateTrackedTag( + device: DictationKeepAwakeDevice, + tag: string, + isStillWanted: () => boolean +): Promise { + const nativeActivation = device.activate(tag) try { await withNativeCallTimeout(nativeActivation) } catch (err) { @@ -80,7 +84,7 @@ async function activateTrackedTag(tag: string, isStillWanted: () => boolean): Pr return } // No owner wants it anymore — the screen must not stay awake. - await deactivateTrackedTag(tag).catch(() => undefined) + await deactivateTrackedTag(device, tag).catch(() => undefined) }), () => { // A late definite rejection means nothing activated after all, but @@ -98,9 +102,9 @@ async function activateTrackedTag(tag: string, isStillWanted: () => boolean): Pr pendingActivations.delete(tag) } -async function deactivateTrackedTag(tag: string): Promise { +async function deactivateTrackedTag(device: DictationKeepAwakeDevice, tag: string): Promise { try { - await withNativeCallTimeout(deactivateKeepAwake(tag)) + await withNativeCallTimeout(device.deactivate(tag)) } catch (err) { // A replacement hook must be able to retry cleanup after Android replaces // an Activity and the owner that acquired this tag has unmounted. @@ -112,7 +116,7 @@ async function deactivateTrackedTag(tag: string): Promise { pendingActivations.delete(tag) } -async function cleanupPendingTags(): Promise { +async function cleanupPendingTags(device: DictationKeepAwakeDevice): Promise { const staleTags = new Set(pendingCleanupTags) for (const [tag, isStillWanted] of pendingActivations) { // A still-wanted timed-out activation is not an orphan: deactivating it @@ -129,7 +133,7 @@ async function cleanupPendingTags(): Promise { // swallow failures: a stale tag that still cannot be deactivated must not // fail the fresh acquire that triggered this retry; it stays queued. await Promise.allSettled( - Array.from(staleTags, (tag) => deactivateTrackedTag(tag).catch(() => undefined)) + Array.from(staleTags, (tag) => deactivateTrackedTag(device, tag).catch(() => undefined)) ) } @@ -137,10 +141,14 @@ export class MobileDictationKeepAwakeOwner { private readonly ownerId = createOwnerId() private acquiredTag: string | null = null + /** The two calls that differ between the hosts, and the only part of this file that does: the + * tag pools, the serialized queue, the timeouts and the retries are the same either way. */ + constructor(private readonly device: DictationKeepAwakeDevice) {} + acquire(dictationId: string): Promise { const tag = this.createTag(dictationId) return enqueueKeepAwakeOperation(async () => { - await cleanupPendingTags() + await cleanupPendingTags(this.device) if (this.acquiredTag && !activeTags.has(this.acquiredTag)) { this.acquiredTag = null } @@ -152,13 +160,13 @@ export class MobileDictationKeepAwakeOwner { this.acquiredTag = null // Best-effort: a failed previous-tag cleanup is queued for retry and // must not block recording intent for the new dictation below. - await deactivateTrackedTag(previousTag).catch(() => undefined) + await deactivateTrackedTag(this.device, previousTag).catch(() => undefined) } // Record ownership before the native call: acquiredTag is intent while // activeTags is native state, so a failed initial activation can still // be healed by a later foreground reacquire. this.acquiredTag = tag - await activateTrackedTag(tag, () => this.acquiredTag === tag) + await activateTrackedTag(this.device, tag, () => this.acquiredTag === tag) }) } @@ -168,7 +176,7 @@ export class MobileDictationKeepAwakeOwner { reacquire(dictationId: string): Promise { const tag = this.createTag(dictationId) return enqueueKeepAwakeOperation(async () => { - await cleanupPendingTags() + await cleanupPendingTags(this.device) if (this.acquiredTag !== tag) { return } @@ -176,7 +184,7 @@ export class MobileDictationKeepAwakeOwner { // re-applies the window flag from an empty tag set — deactivate both. if (activeTags.has(tag) || pendingActivations.has(tag)) { try { - await deactivateTrackedTag(tag) + await deactivateTrackedTag(this.device, tag) } catch (err) { // A still-live tag must not sit in the orphan pool where another // owner's drain would turn it off without reactivating; keep it in @@ -192,7 +200,7 @@ export class MobileDictationKeepAwakeOwner { // Known gap: if another expo-keep-awake owner exists (e.g. dev-build // dev tools), the native module never empties its tag set, so the // deactivate/activate cycle cannot re-apply the Android window flag. - await activateTrackedTag(tag, () => this.acquiredTag === tag) + await activateTrackedTag(this.device, tag, () => this.acquiredTag === tag) }) } @@ -208,14 +216,14 @@ export class MobileDictationKeepAwakeOwner { this.acquiredTag = null return } - await deactivateTrackedTag(tag) + await deactivateTrackedTag(this.device, tag) this.acquiredTag = null } finally { // Drain after the owner-local unset so this owner's own timed-out // activation is no longer wanted and gets cleaned here — an acquire // may never happen again this session. Still-wanted tags of other // live dictations are spared by the drain itself. - await cleanupPendingTags() + await cleanupPendingTags(this.device) } }) } @@ -225,12 +233,16 @@ export class MobileDictationKeepAwakeOwner { } } -export function createMobileDictationKeepAwakeOwner(): MobileDictationKeepAwakeOwner { - return new MobileDictationKeepAwakeOwner() +export function createMobileDictationKeepAwakeOwner( + device: DictationKeepAwakeDevice +): MobileDictationKeepAwakeOwner { + return new MobileDictationKeepAwakeOwner(device) } // Foreground is the retry point for wake tags whose final deactivation timed // out after a dictation ended — otherwise nothing runs until the next one. -export function drainMobileDictationKeepAwakeCleanup(): Promise { - return enqueueKeepAwakeOperation(cleanupPendingTags) +export function drainMobileDictationKeepAwakeCleanup( + device: DictationKeepAwakeDevice +): Promise { + return enqueueKeepAwakeOperation(() => cleanupPendingTags(device)) } diff --git a/mobile/src/hooks/mobile-dictation-stop-tail.test.tsx b/mobile/src/hooks/mobile-dictation-stop-tail.test.tsx new file mode 100644 index 00000000000..c92323a9688 --- /dev/null +++ b/mobile/src/hooks/mobile-dictation-stop-tail.test.tsx @@ -0,0 +1,193 @@ +/** + * The ordering `stop()` depends on: the capture hands over its tail, and only then does the flow + * stop accepting chunks. + * + * On the page the tail is real audio — up to one drain interval of what the user was still saying + * as they lifted the button, fetched by the last read inside `end()`. Refusing chunks first drops + * exactly that, and taking the pending set before it lets `finish` overtake the last send. Neither + * shows up in a source-text check: both orders put `end()` before `Promise.allSettled`, and + * reversing the two lines left the whole mobile suite green. + * + * Driven against a capture whose `end()` delivers a chunk, which is what the page's seam does and + * what the device's never does, so this is the one case that can tell the orders apart. + */ +import { createElement } from 'react' +import { act, create } from 'react-test-renderer' +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { + createFakeRpcClient, + type FakeRpcClient, + type SentRequest +} from '../mobile-web-shell/bridge-host-test-fakes' +import type { + DictationCapture, + DictationCaptureChunk +} from '../platform/dictation-capture-contract' + +type Seam = { + chunkHandlers: Set<(chunk: DictationCaptureChunk) => void> + /** Bytes the capture is still holding when `end()` is called, as the shell's ring would be. */ + tail: Uint8Array | null +} + +const seam = vi.hoisted((): Seam => ({ chunkHandlers: new Set(), tail: null })) + +vi.mock('react-native', () => ({ + AppState: { currentState: 'active', addEventListener: () => ({ remove: () => {} }) }, + Platform: { OS: 'ios' } +})) + +// One object for the life of the module, because the hook keys its effects on the capture's +// identity: the teardown effect runs whenever it changes, so a seam returning a fresh object per +// render would cancel the dictation on every render. Both real seams are stable — the native one is +// a module const, the page's is a `useMemo` on the client. +vi.mock('../platform/dictation-capture', () => { + const capture: DictationCapture = { + open: async () => ({ ok: true }), + begin: () => true, + end: async () => { + // The page's seam reads once more here, and that read can carry audio. + const tail = seam.tail + seam.tail = null + if (tail !== null) { + for (const handler of seam.chunkHandlers) { + handler({ data: tail, droppedBytes: 0 }) + } + } + }, + release: () => {}, + onChunk: (handler) => { + seam.chunkHandlers.add(handler) + return { + remove: () => { + seam.chunkHandlers.delete(handler) + } + } + }, + onInterruption: () => ({ remove: () => {} }), + keepAwake: { activate: async () => {}, deactivate: async () => {} } + } + return { useDictationCapture: () => capture } +}) + +import { useMobileDictation, type UseMobileDictationResult } from './use-mobile-dictation' + +/** The desktop, answering whatever the hook forwards. `finish` carries the transcript, which is + * the one reply the flow reads. */ +function settle(rpc: FakeRpcClient, sent: SentRequest[]): void { + for (const request of rpc.requests.splice(0)) { + sent.push(request) + request.resolve({ + id: 'desktop', + ok: true, + result: request.method === 'speech.dictation.finish' ? { text: 'a sentence' } : {} + }) + } +} + +/** The base64 a chunk request carried, read by narrowing rather than asserted: the fake records + * whatever the hook passed, and this is a test of what that was. */ +/** Drains and answers whatever the hook sends, for as long as it keeps sending: one dictation is a + * chain of requests where each is only made once the one before it settled. */ +async function pump(rpc: FakeRpcClient, sent: SentRequest[]): Promise { + for (let round = 0; round < 8; round += 1) { + settle(rpc, sent) + await Promise.resolve() + await Promise.resolve() + } +} + +function audioOf(request: SentRequest): unknown { + const params = request.args[1] + return typeof params === 'object' && params !== null && 'audioBase64' in params + ? params.audioBase64 + : null +} + +const held: { dictation: UseMobileDictationResult | null } = { dictation: null } + +function mount(client: FakeRpcClient): void { + function Probe(): null { + held.dictation = useMobileDictation({ + client, + enabled: true, + onTranscript: () => {}, + onError: () => {} + }) + return null + } + act(() => { + create(createElement(Probe)) + }) +} + +function dictation(): UseMobileDictationResult { + const current = held.dictation + if (current === null) { + throw new Error('nothing mounted') + } + return current +} + +beforeEach(() => { + seam.chunkHandlers.clear() + seam.tail = null + held.dictation = null +}) + +describe('the audio a capture hands over as it ends', () => { + it('is still accepted, and reaches the desktop before the finish', async () => { + const rpc = createFakeRpcClient() + const sent: SentRequest[] = [] + mount(rpc) + await act(async () => { + const started = dictation().start() + await pump(rpc, sent) + await started + }) + // What the user was still saying when they lifted the button, which no timer will come for. + seam.tail = Uint8Array.from([7, 8, 9, 10]) + await act(async () => { + const stopped = dictation().stop() + await pump(rpc, sent) + await stopped + }) + const methods = sent.map((request) => request.method) + expect(methods).toContain('speech.dictation.chunk') + // Refusing chunks before `end()` drops this one silently: the handler reads + // `acceptingChunksRef` and returns, and the transcript loses the end of the sentence. + const chunk = sent.find((request) => request.method === 'speech.dictation.chunk') + expect(chunk === undefined ? null : audioOf(chunk)).toBe('BwgJCg==') + // And it is sent before the finish, or the desktop transcribes without it. + expect(methods.indexOf('speech.dictation.chunk')).toBeLessThan( + methods.indexOf('speech.dictation.finish') + ) + }) + + it('stops being accepted once the capture has ended', async () => { + const rpc = createFakeRpcClient() + const sent: SentRequest[] = [] + mount(rpc) + await act(async () => { + const started = dictation().start() + await pump(rpc, sent) + await started + }) + await act(async () => { + const stopped = dictation().stop() + await pump(rpc, sent) + await stopped + }) + const before = sent.filter((request) => request.method === 'speech.dictation.chunk').length + // A late event from a capture that has already ended is not this dictation's audio. + for (const handler of seam.chunkHandlers) { + handler({ data: Uint8Array.from([1, 2, 3, 4]), droppedBytes: 0 }) + } + await act(async () => { + await pump(rpc, sent) + }) + expect(sent.filter((request) => request.method === 'speech.dictation.chunk')).toHaveLength( + before + ) + }) +}) diff --git a/mobile/src/hooks/use-mobile-dictation-source.test.ts b/mobile/src/hooks/use-mobile-dictation-source.test.ts index 3b1e305d1ae..7354c2019bf 100644 --- a/mobile/src/hooks/use-mobile-dictation-source.test.ts +++ b/mobile/src/hooks/use-mobile-dictation-source.test.ts @@ -22,6 +22,10 @@ const foregroundKeepAwakeSource = readFileSync( new URL('./mobile-dictation-foreground-keep-awake.ts', import.meta.url), 'utf8' ) +const nativeCaptureSource = readFileSync( + new URL('../platform/dictation-capture.ts', import.meta.url), + 'utf8' +) function sliceSource(sourceText: string, startPattern: string, endPattern: string): string { const start = sourceText.indexOf(startPattern) @@ -69,18 +73,29 @@ describe('useMobileDictation source invariants', () => { expect(reserveIndex).toBeLessThan(encodeIndex) expect(microphoneEffect).toContain('MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE') expect(microphoneEffect).toContain('queue.pendingAudioBudget.release(byteLength)') - expect(source).toContain('enqueueMobileDictationAudioChunk(client, dictationId, event') + expect(source).toContain('enqueueMobileDictationAudioChunk(client, dictationId, chunk') + }) + + it('carries audio the capture dropped into the same refusal the budget raises', () => { + const chunkHandler = sliceBetween('const sub = capture.onChunk(', 'return () => sub.remove()') + expect(chunkHandler).toContain('if (chunk.droppedBytes > 0)') + expect(chunkHandler).toContain('MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE') + // Only the page can drop: the seam's native half is where the microphone is. + expect(nativeCaptureSource).toContain('droppedBytes: 0') }) it('reuses audio chunk queue wiring across microphone events', () => { const queueIndex = source.indexOf('const audioChunkQueue =') - const listenerIndex = source.indexOf("addExpoTwoWayAudioEventListener('onMicrophoneData'") + const listenerIndex = source.indexOf('capture.onChunk(') expect(queueIndex).toBeGreaterThanOrEqual(0) expect(queueIndex).toBeLessThan(listenerIndex) expect(source).toContain( - 'enqueueMobileDictationAudioChunk(client, dictationId, event, audioChunkQueue)' + 'enqueueMobileDictationAudioChunk(client, dictationId, chunk, audioChunkQueue)' ) + // The native half is the same calls in the same order it always made them; what moved is where + // they are written, so the page can answer the same shape. + expect(nativeCaptureSource).toContain("addExpoTwoWayAudioEventListener('onMicrophoneData'") }) it('resets pending audio bytes whenever pending chunk tracking is cleared', () => { @@ -94,12 +109,16 @@ describe('useMobileDictation source invariants', () => { expect(source).toMatch( /import \{[^}]*createMobileDictationKeepAwakeOwner[^}]*\} from '\.\/mobile-dictation-keep-awake'/ ) - expect(source).toContain( - 'const keepAwakeOwner = useMemo(() => createMobileDictationKeepAwakeOwner(), [])' - ) - expect(keepAwakeSource).toContain('activateKeepAwakeAsync') - expect(keepAwakeSource).toContain('deactivateKeepAwake') - expect(keepAwakeSource).not.toMatch(/\bactivateKeepAwake\s*\(/) + expect(source).toContain('createMobileDictationKeepAwakeOwner(capture.keepAwake)') + // The tag bookkeeping is host-independent and holds no device of its own: the two calls that + // differ come in through the seam, which is `expo-keep-awake` natively and the shell's wake + // verb on the page. + expect(keepAwakeSource).not.toMatch(/from '(expo-keep-awake|.*two-way-audio)'/) + expect(keepAwakeSource).toContain('device.activate(tag)') + expect(keepAwakeSource).toContain('device.deactivate(tag)') + expect(nativeCaptureSource).toContain('activateKeepAwakeAsync') + expect(nativeCaptureSource).toContain('deactivateKeepAwake') + expect(nativeCaptureSource).not.toMatch(/\bactivateKeepAwake\s*\(/) }) it('acquires keep-awake only after desktop start and stale-start guards', () => { @@ -113,7 +132,7 @@ describe('useMobileDictation source invariants', () => { ) const acquireIndex = startBody.indexOf('.acquire(dictationId)') const desktopSessionIndex = hookStartBody.indexOf('await startMobileDictationDesktopSession') - const toggleRecordingIndex = hookStartBody.indexOf('toggleRecording(true)') + const toggleRecordingIndex = hookStartBody.indexOf('capture.begin()') expect(desktopStartIndex).toBeGreaterThanOrEqual(0) expect(acquireIndex).toBeGreaterThan(desktopStartIndex) @@ -181,7 +200,7 @@ describe('useMobileDictation source invariants', () => { 'const closeDictationAudio = useCallback(', 'const failActiveDictation =' ) - expect(closeAudio.indexOf('toggleRecording(false)')).toBeLessThan( + expect(closeAudio.indexOf('capture.end()')).toBeLessThan( closeAudio.indexOf('void keepAwakeOwner.release') ) expect(closeAudio).toContain('.catch(() => undefined)') @@ -197,7 +216,7 @@ describe('useMobileDictation source invariants', () => { } const stopBody = sliceBetween('const stop = useCallback(async () => {', 'const cancel =') - expect(stopBody.indexOf('toggleRecording(false)')).toBeLessThan( + expect(stopBody.indexOf('capture.end()')).toBeLessThan( stopBody.indexOf('await Promise.allSettled') ) // The wake tag must be held through chunk drain and the finish RPC so a @@ -211,15 +230,19 @@ describe('useMobileDictation source invariants', () => { }) it('reacquires the wake tag when Android returns to the foreground mid-dictation', () => { - expect(source).toContain('useMobileDictationForegroundKeepAwake(keepAwakeOwner, activeIdRef)') + expect(source).toContain( + 'useMobileDictationForegroundKeepAwake(keepAwakeOwner, activeIdRef, capture.keepAwake)' + ) expect(foregroundKeepAwakeSource).toContain("Platform.OS !== 'android'") expect(foregroundKeepAwakeSource).toContain('keepAwakeOwner.reacquire(dictationId)') // A transiently failing refresh retries while the dictation is live. expect(foregroundKeepAwakeSource).toContain('REACQUIRE_RETRY_DELAYS_MS[attempt]') expect(foregroundKeepAwakeSource).toContain('activeIdRef.current === dictationId') // Stale-tag retries survive hook unmount via a module-level listener. - expect(foregroundKeepAwakeSource).toContain('installGlobalStaleTagForegroundDrain()') - expect(foregroundKeepAwakeSource).toContain('drainMobileDictationKeepAwakeCleanup()') + expect(foregroundKeepAwakeSource).toContain( + 'installGlobalStaleTagForegroundDrain(keepAwakeDevice)' + ) + expect(foregroundKeepAwakeSource).toContain('drainMobileDictationKeepAwakeCleanup(current)') // Native activate skips re-applying the window flag while any tag remains, // so reacquire must deactivate before activating. @@ -228,10 +251,14 @@ describe('useMobileDictation source invariants', () => { 'reacquire(dictationId: string)', 'release(dictationId?: string)' ) - expect(reacquireBody.indexOf('await activateTrackedTag(tag,')).toBeGreaterThanOrEqual(0) - expect(reacquireBody.indexOf('deactivateTrackedTag(tag)')).toBeGreaterThanOrEqual(0) - expect(reacquireBody.indexOf('deactivateTrackedTag(tag)')).toBeLessThan( - reacquireBody.indexOf('await activateTrackedTag(tag,') + expect( + reacquireBody.indexOf('await activateTrackedTag(this.device, tag,') + ).toBeGreaterThanOrEqual(0) + expect(reacquireBody.indexOf('deactivateTrackedTag(this.device, tag)')).toBeGreaterThanOrEqual( + 0 + ) + expect(reacquireBody.indexOf('deactivateTrackedTag(this.device, tag)')).toBeLessThan( + reacquireBody.indexOf('await activateTrackedTag(this.device, tag,') ) }) @@ -240,33 +267,40 @@ describe('useMobileDictation source invariants', () => { 'const closeDictationAudio = useCallback(', 'const failActiveDictation =' ) - const toggleIndex = closeAudio.indexOf('toggleRecording(false)') + const toggleIndex = closeAudio.indexOf('capture.end()') const catchIndex = closeAudio.indexOf('} catch', toggleIndex) const releaseIndex = closeAudio.indexOf('void keepAwakeOwner.release') expect(toggleIndex).toBeGreaterThanOrEqual(0) expect(catchIndex).toBeGreaterThan(toggleIndex) expect(catchIndex).toBeLessThan(releaseIndex) + // The try above is not what makes this true, and this case used to claim it was. `end` is + // async, so a throwing binding rejects rather than throwing, and a synchronous `catch` around + // `void capture.end()` never sees it. The guard is the seam swallowing its own failure, which + // `dictation-capture.test.ts` drives against an engine that will not stop; the try stays for a + // seam that throws synchronously. + expect(nativeCaptureSource).toContain("console.error('Failed to stop microphone recording'") + expect(nativeCaptureSource).toContain("console.error('Failed to tear down the audio session'") + // stop()'s recording shutdown sits inside the try so a native throw still // runs the finally release and error cleanup. const stopBody = sliceBetween('const stop = useCallback(async () => {', 'const cancel =') expect(stopBody.indexOf('try {')).toBeGreaterThanOrEqual(0) - expect(stopBody.indexOf('try {')).toBeLessThan(stopBody.indexOf('toggleRecording(false)')) + expect(stopBody.indexOf('try {')).toBeLessThan(stopBody.indexOf('capture.end()')) }) it('routes disabled state and audio interruptions through cancel cleanup', () => { - const interruptionEffect = sliceBetween( - "addExpoTwoWayAudioEventListener('onAudioInterruption'", - 'return () => sub.remove()' - ) + const interruptionEffect = sliceBetween('capture.onInterruption(', 'return () => sub.remove()') const disabledEffect = sliceBetween( 'useEffect(() => {\n if (!enabled) {', ' }, [cancel, enabled])' ) - expect(interruptionEffect).toContain("event.data === 'began' || event.data === 'blocked'") expect(interruptionEffect).toContain('void cancel()') expect(disabledEffect).toContain('void cancel()') + // Which interruptions end a capture is one predicate both seams read, so a page cannot cancel + // on a kind the device ignores. `dictation-capture.test.ts` drives the rule itself. + expect(nativeCaptureSource).toContain('bridgeAudioInterruptionEndsCapture(event.data)') }) it('uses per-owner dictation keep-awake tags and serializes async ownership changes', () => { @@ -281,6 +315,6 @@ describe('useMobileDictation source invariants', () => { 'const targetTag = dictationId ? this.createTag(dictationId) : null' ) expect(keepAwakeSource).toContain('if (!tag || (targetTag && tag !== targetTag))') - expect(keepAwakeSource).toContain('await cleanupPendingTags()') + expect(keepAwakeSource).toContain('await cleanupPendingTags(this.device)') }) }) diff --git a/mobile/src/hooks/use-mobile-dictation.ts b/mobile/src/hooks/use-mobile-dictation.ts index e835592d685..733d259e5ba 100644 --- a/mobile/src/hooks/use-mobile-dictation.ts +++ b/mobile/src/hooks/use-mobile-dictation.ts @@ -1,12 +1,9 @@ import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react' +import { useDictationCapture } from '../platform/dictation-capture' import { - addExpoTwoWayAudioEventListener, - initialize, - requestMicrophonePermissionsAsync, - tearDown, - toggleRecording -} from '@orca/expo-two-way-audio' -import { MobileDictationPendingAudioBudget } from './mobile-dictation-pending-audio-budget' + MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE, + MobileDictationPendingAudioBudget +} from './mobile-dictation-pending-audio-budget' import { enqueueMobileDictationAudioChunk } from './mobile-dictation-audio-chunk' import { createMobileDictationKeepAwakeOwner } from './mobile-dictation-keep-awake' import { useMobileDictationForegroundKeepAwake } from './mobile-dictation-foreground-keep-awake' @@ -31,7 +28,13 @@ export type { UseMobileDictationResult } from './mobile-dictation-session-state' export function useMobileDictation(options: UseMobileDictationOptions): UseMobileDictationResult { const { client, enabled, onTranscript, onError } = options - const keepAwakeOwner = useMemo(() => createMobileDictationKeepAwakeOwner(), []) + // One seam, two hosts: natively the microphone and `expo-keep-awake`, on the page the shell's + // four verbs. Everything below this line is the same flow either way. + const capture = useDictationCapture() + const keepAwakeOwner = useMemo( + () => createMobileDictationKeepAwakeOwner(capture.keepAwake), + [capture] + ) const [status, setStatus] = useState('idle') const [error, setError] = useState(null) const activeIdRef = useRef(null) @@ -67,7 +70,7 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil pendingChunksRef.current.clear() pendingAudioBudgetRef.current.reset() try { - toggleRecording(false) + void capture.end() } catch (err) { // Cleanup must keep going when native recording shutdown throws, or // the wake tag and dictation state would leak. @@ -75,7 +78,7 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil } void keepAwakeOwner.release(dictationId ?? undefined).catch(() => undefined) }, - [keepAwakeOwner] + [capture, keepAwakeOwner] ) const failActiveDictation = useCallback( @@ -104,16 +107,23 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil activeIdRef.current === id || finishingIdRef.current === id, failActiveDictation } - const sub = addExpoTwoWayAudioEventListener('onMicrophoneData', (event) => { + const sub = capture.onChunk((chunk) => { const client = clientRef.current const dictationId = activeIdRef.current if (!client || !dictationId || !enabledRef.current || !acceptingChunksRef.current) { return } - enqueueMobileDictationAudioChunk(client, dictationId, event, audioChunkQueue) + if (chunk.droppedBytes > 0) { + // Audio the capture already lost is the condition the budget refuses on by another route — + // the page is not keeping up with the microphone — so it reaches the one message the + // composer renders for it. Only the page can drop: natively this is always zero. + failActiveDictation(dictationId, new Error(MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE)) + return + } + enqueueMobileDictationAudioChunk(client, dictationId, chunk, audioChunkQueue) }) return () => sub.remove() - }, [failActiveDictation, reportError]) + }, [capture, failActiveDictation, reportError]) const start = useCallback(async () => { const client = clientRef.current @@ -125,29 +135,30 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil generationRef.current = generation setError(null) setStatus('starting') - const permission = await requestMicrophonePermissionsAsync() + let opened + try { + opened = await capture.open() + } catch (err) { + // A capture the host refused outright, which on the page is a route that was never granted + // the audio verbs. Back to idle before it is rethrown: the caller toasts the shell's own + // message, and a control left on 'starting' has no way back short of a remount. + setStatus('idle') + throw err instanceof Error ? err : new Error(String(err)) + } if (generationRef.current !== generation || !enabledRef.current) { + capture.release() if (generationRef.current === generation) { setStatus('idle') } return } - if (!permission.granted) { + if (!opened.ok) { setStatus('idle') - throw new Error('Microphone permission denied') - } - - const initialized = await initialize() - if (generationRef.current !== generation || !enabledRef.current) { - void tearDown() - if (generationRef.current === generation) { - setStatus('idle') - } - return - } - if (!initialized) { - setStatus('idle') - throw new Error('Failed to initialize microphone') + throw new Error( + opened.reason === 'permission-denied' + ? 'Microphone permission denied' + : 'Failed to initialize microphone' + ) } const dictationId = createMobileDictationId() @@ -171,7 +182,7 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil acceptingChunksRef.current = true pendingChunksRef.current.clear() pendingAudioBudgetRef.current.reset() - if (!toggleRecording(true)) { + if (!capture.begin()) { return false } setStatus('recording') @@ -181,10 +192,10 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil acceptingChunksRef.current = false pendingChunksRef.current.clear() pendingAudioBudgetRef.current.reset() - toggleRecording(false) + void capture.end() } }) - }, [keepAwakeOwner]) + }, [capture, keepAwakeOwner]) const stop = useCallback(async () => { const client = clientRef.current @@ -197,11 +208,16 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil generationRef.current = generation finishingIdRef.current = dictationId setStatus('processing') - acceptingChunksRef.current = false try { // Inside the try so a throwing native shutdown still runs the finally // release and error cleanup. - toggleRecording(false) + // + // Awaited, and chunks are still accepted while it runs: `end` hands over whatever the + // capture is still holding, which on the page is up to one drain interval of the tail of + // what the user just said. Refusing chunks first would drop exactly that audio, and taking + // the pending set before it would let `finish` overtake the last send. + await capture.end() + acceptingChunksRef.current = false await Promise.allSettled(Array.from(pendingChunksRef.current)) if ( !isCurrentMobileDictationFinish( @@ -256,7 +272,7 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil finishingIdRef.current = null } } - }, [failActiveDictation, keepAwakeOwner]) + }, [capture, failActiveDictation, keepAwakeOwner]) const cancel = useCallback(async () => { const client = clientRef.current @@ -272,16 +288,14 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil setError(null) }, [closeDictationAudio]) - useMobileDictationForegroundKeepAwake(keepAwakeOwner, activeIdRef) + useMobileDictationForegroundKeepAwake(keepAwakeOwner, activeIdRef, capture.keepAwake) useEffect(() => { - const sub = addExpoTwoWayAudioEventListener('onAudioInterruption', (event) => { - if (event.data === 'began' || event.data === 'blocked') { - void cancel() - } + const sub = capture.onInterruption(() => { + void cancel() }) return () => sub.remove() - }, [cancel]) + }, [cancel, capture]) useEffect(() => { if (!enabled) { @@ -296,14 +310,14 @@ export function useMobileDictation(options: UseMobileDictationOptions): UseMobil activeIdRef.current = null finishingIdRef.current = null closeDictationAudio(dictationId) - void tearDown() + capture.release() if (clientRef.current && dictationId) { void dictationSessionCancel .request(clientRef.current, { dictationId }) .catch(() => undefined) } } - }, [closeDictationAudio]) + }, [capture, closeDictationAudio]) return { status, diff --git a/mobile/src/mobile-web-shell/MobileWebShellScreen.test.tsx b/mobile/src/mobile-web-shell/MobileWebShellScreen.test.tsx index 7f6958ec4f4..34271ce508d 100644 --- a/mobile/src/mobile-web-shell/MobileWebShellScreen.test.tsx +++ b/mobile/src/mobile-web-shell/MobileWebShellScreen.test.tsx @@ -90,6 +90,18 @@ vi.mock('expo-haptics', () => ({ NotificationFeedbackType: { Error: 'error', Success: 'success' } })) vi.mock('expo-document-picker', () => ({ getDocumentAsync: () => Promise.resolve(null) })) +vi.mock('@orca/expo-two-way-audio', () => ({ + addExpoTwoWayAudioEventListener: () => ({ remove: () => {} }), + initialize: () => Promise.resolve(true), + requestMicrophonePermissionsAsync: () => + Promise.resolve({ granted: true, canAskAgain: true, status: 'granted', expires: 'never' }), + tearDown: () => {}, + toggleRecording: () => true +})) +vi.mock('expo-keep-awake', () => ({ + activateKeepAwakeAsync: () => Promise.resolve(), + deactivateKeepAwake: () => Promise.resolve() +})) vi.mock('expo-image-picker', () => ({ launchImageLibraryAsync: () => Promise.resolve({ canceled: true }), requestMediaLibraryPermissionsAsync: () => Promise.resolve({ granted: false }) diff --git a/mobile/src/mobile-web-shell/bridge-host-errors.ts b/mobile/src/mobile-web-shell/bridge-host-errors.ts index 4f195e5da93..99940563c56 100644 --- a/mobile/src/mobile-web-shell/bridge-host-errors.ts +++ b/mobile/src/mobile-web-shell/bridge-host-errors.ts @@ -74,7 +74,11 @@ export const BRIDGE_NATIVE_REFUSAL_CODES = [ * expo-image-picker 55.0.24 — and it is kept for the first that does. */ 'native_media_permission_denied', /** A `native.` method on a `subscribe`, which this seam answers on requests only. */ - 'native_verb_not_a_stream' + 'native_verb_not_a_stream', + /** A read or a wake-tag call for a capture this session does not have: never started, stopped, + * or ended with the page that asked for it. One code for all three, because the page's answer + * to each is the same — its capture is over and its dictation with it. */ + 'native_audio_not_capturing' ] as const export type BridgeNativeRefusalCode = (typeof BRIDGE_NATIVE_REFUSAL_CODES)[number] diff --git a/mobile/src/mobile-web-shell/bridge-host-init.test.ts b/mobile/src/mobile-web-shell/bridge-host-init.test.ts index dbb116b60d1..d46ce232614 100644 --- a/mobile/src/mobile-web-shell/bridge-host-init.test.ts +++ b/mobile/src/mobile-web-shell/bridge-host-init.test.ts @@ -48,7 +48,11 @@ describe('init and state', () => { 'native.clipboard.read', 'native.media.pick', 'native.media.read', - 'native.media.release' + 'native.media.release', + 'native.audio.start', + 'native.audio.read', + 'native.audio.stop', + 'native.wakelock.set' ] }, route: ROUTE, diff --git a/mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.test.ts b/mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.test.ts new file mode 100644 index 00000000000..24734090f64 --- /dev/null +++ b/mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.test.ts @@ -0,0 +1,622 @@ +/** + * The four audio verbs: what their schemas refuse, and what the shell's capture answers. + * + * The handler is driven through its engine seam rather than through `@orca/expo-two-way-audio`, + * for the reason the media verbs' device half is driven through one: the arms worth pinning — a + * denied microphone, a ring that filled, a read after the capture ended — are exactly the ones a + * simulator makes expensive, and none of them is a fact about Swift. + */ +import { describe, expect, it } from 'vitest' +import { MobileWebBundleRouteSchema } from '../../../../src/shared/mobile-web-bundle/manifest-contract' +import { MOBILE_DICTATION_MAX_PENDING_AUDIO_BYTES } from '../../hooks/mobile-dictation-pending-audio-budget' +import { BridgeNativeVerbRefusedError } from '../bridge-host-errors' +import { createNativeAudioCapture, type NativeAudioEngine } from '../../platform/native-audio' +import { createNativeWakelockServer } from '../../platform/native-wakelock' +import { + BRIDGE_AUDIO_READ_MAX_BASE64_CHARS, + BRIDGE_AUDIO_RING_MAX_BYTES, + audioReadParamsSchema, + audioReadResultSchema, + audioStartParamsSchema, + audioStopParamsSchema, + wakelockSetParamsSchema +} from './bridge-audio-verbs' +import { BRIDGE_NATIVE_VERB_NAMES, BRIDGE_NATIVE_VERBS } from './bridge-native-verbs' + +const AUDIO_VERBS = [ + 'native.audio.start', + 'native.audio.read', + 'native.audio.stop', + 'native.wakelock.set' +] as const + +/** An engine whose every call is a value a case can set, and whose events a case can fire. */ +function createTestEngine( + overrides: Partial<{ + permission: NativeAudioEngine['requestPermission'] + open: NativeAudioEngine['open'] + begin: NativeAudioEngine['begin'] + }> = {} +) { + const microphone: ((bytes: Uint8Array) => void)[] = [] + const interruptions: ((kind: 'began' | 'ended' | 'blocked') => void)[] = [] + const log: string[] = [] + const engine: NativeAudioEngine = { + requestPermission: overrides.permission ?? (async () => 'granted'), + open: overrides.open ?? (async (sampleRate) => ({ opened: true, sampleRate })), + begin: overrides.begin ?? (() => true), + end: () => { + log.push('end') + }, + onMicrophoneData: (handler) => { + microphone.push(handler) + return { + remove: () => { + microphone.splice(microphone.indexOf(handler), 1) + log.push('microphone-off') + } + } + }, + onInterruption: (handler) => { + interruptions.push(handler) + return { + remove: () => { + interruptions.splice(interruptions.indexOf(handler), 1) + } + } + } + } + return { + engine, + log, + /** How many handlers the engine is still calling. One per live capture, or a leak. */ + liveListeners: () => ({ microphone: microphone.length, interruptions: interruptions.length }), + emit: (bytes: Uint8Array) => { + for (const handler of microphone) { + handler(bytes) + } + }, + interrupt: (kind: 'began' | 'ended' | 'blocked') => { + for (const handler of interruptions) { + handler(kind) + } + } + } +} + +function pcm(byteLength: number, seed = 0): Uint8Array { + const bytes = new Uint8Array(byteLength) + for (let index = 0; index < byteLength; index += 1) { + bytes[index] = (index * 31 + seed) % 251 + } + return bytes +} + +function decode(base64: string): Uint8Array { + const binary = atob(base64) + const bytes = new Uint8Array(binary.length) + for (let index = 0; index < binary.length; index += 1) { + bytes[index] = binary.charCodeAt(index) + } + return bytes +} + +describe('the audio verbs in the table', () => { + it('lists all four, each under a name a manifest grant may carry', () => { + for (const verb of AUDIO_VERBS) { + expect(BRIDGE_NATIVE_VERB_NAMES, verb).toContain(verb) + expect(BRIDGE_NATIVE_VERBS[verb], verb).toBeDefined() + // The ruling-6a trap, pinned against the schema itself rather than against a copy of its + // regex: `native.audio.readChunk` is not a route that falls back to native, it is a bundle + // the phone refuses entire. + expect( + MobileWebBundleRouteSchema.safeParse({ pathname: '/h', grants: [verb] }).success, + verb + ).toBe(true) + } + }) + + it('refuses the camel-cased spelling of the read, which is what makes the name load-bearing', () => { + expect( + MobileWebBundleRouteSchema.safeParse({ pathname: '/h', grants: ['native.audio.readChunk'] }) + .success + ).toBe(false) + }) +}) + +describe('what the audio schemas refuse', () => { + it('refuses a start with no rate, a rate off the grid, and an unknown param', () => { + expect(audioStartParamsSchema.safeParse({}).success).toBe(false) + expect(audioStartParamsSchema.safeParse({ sampleRate: 16_000.5 }).success).toBe(false) + expect(audioStartParamsSchema.safeParse({ sampleRate: 96_000 }).success).toBe(false) + expect(audioStartParamsSchema.safeParse({ sampleRate: 0 }).success).toBe(false) + expect(audioStartParamsSchema.safeParse({ sampleRate: 16_000, channels: 1 }).success).toBe( + false + ) + expect(audioStartParamsSchema.safeParse({ sampleRate: 16_000 }).success).toBe(true) + }) + + it("holds a read to the ring, which is the page's own pending-audio budget", () => { + expect(BRIDGE_AUDIO_RING_MAX_BYTES).toBe(MOBILE_DICTATION_MAX_PENDING_AUDIO_BYTES) + expect(audioReadParamsSchema.safeParse({ maxBytes: 0 }).success).toBe(false) + expect( + audioReadParamsSchema.safeParse({ maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES + 1 }).success + ).toBe(false) + expect(audioReadParamsSchema.safeParse({ maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }).success).toBe( + true + ) + }) + + it('refuses a stop carrying anything and a wakelock with no tag', () => { + expect(audioStopParamsSchema.safeParse({ why: 'done' }).success).toBe(false) + expect(audioStopParamsSchema.safeParse({}).success).toBe(true) + expect(wakelockSetParamsSchema.safeParse({ active: true }).success).toBe(false) + expect(wakelockSetParamsSchema.safeParse({ active: true, tag: '' }).success).toBe(false) + expect(wakelockSetParamsSchema.safeParse({ active: true, tag: 'x'.repeat(161) }).success).toBe( + false + ) + expect(wakelockSetParamsSchema.safeParse({ active: true, tag: 'orca' }).success).toBe(true) + }) + + it('declares a base64 field a full drain still fits in', () => { + expect(BRIDGE_AUDIO_READ_MAX_BASE64_CHARS).toBe(Math.ceil(BRIDGE_AUDIO_RING_MAX_BYTES / 3) * 4) + expect( + audioReadResultSchema.safeParse({ + base64: 'A'.repeat(BRIDGE_AUDIO_READ_MAX_BASE64_CHARS + 1), + droppedBytes: 0, + recording: true, + interruption: null + }).success + ).toBe(false) + }) +}) + +describe('the shell capture', () => { + it('surfaces a denied microphone as data rather than as a throw', async () => { + const { engine, log } = createTestEngine({ permission: async () => 'denied' }) + const capture = createNativeAudioCapture(engine) + await expect(capture.serve('native.audio.start', { sampleRate: 16_000 })).resolves.toEqual({ + started: false, + sampleRate: 16_000, + permission: 'denied' + }) + // Nothing was opened, so nothing has to be torn down. + expect(log).toEqual([]) + }) + + it('surfaces an engine that would not open on a granted microphone', async () => { + const { engine } = createTestEngine({ + open: async () => ({ opened: false, sampleRate: 16_000 }) + }) + const capture = createNativeAudioCapture(engine) + await expect(capture.serve('native.audio.start', { sampleRate: 16_000 })).resolves.toEqual({ + started: false, + sampleRate: 16_000, + permission: 'granted' + }) + }) + + it('answers the rate the device opened at, not the one that was asked for', async () => { + const { engine } = createTestEngine({ + open: async () => ({ opened: true, sampleRate: 48_000 }) + }) + const capture = createNativeAudioCapture(engine) + await expect(capture.serve('native.audio.start', { sampleRate: 16_000 })).resolves.toEqual({ + started: true, + sampleRate: 48_000, + permission: 'granted' + }) + }) + + it('drains what the microphone produced, in order, and reports no drop', async () => { + const { engine, emit } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + emit(pcm(1_024, 1)) + emit(pcm(1_024, 2)) + const read = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(read.droppedBytes).toBe(0) + expect(read.recording).toBe(true) + expect(read.interruption).toBeNull() + expect(Array.from(decode(read.base64))).toEqual([ + ...Array.from(pcm(1_024, 1)), + ...Array.from(pcm(1_024, 2)) + ]) + }) + + it('serves a partial drain from the front and keeps the rest for the next read', async () => { + const { engine, emit } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + emit(pcm(3_000, 5)) + const first = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: 1_200 }) + ) + const second = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(decode(first.base64).byteLength).toBe(1_200) + expect(decode(second.base64).byteLength).toBe(1_800) + expect(Array.from(decode(second.base64))).toEqual(Array.from(pcm(3_000, 5).subarray(1_200))) + }) + + it('rings at the budget and answers what it could not hold', async () => { + const { engine, emit } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + // One byte short of the ring, then a chunk that cannot fit: the newcomer is dropped, so what + // the page drains is still contiguous audio and never a splice of two moments. + emit(pcm(BRIDGE_AUDIO_RING_MAX_BYTES - 1, 3)) + emit(pcm(64, 4)) + const read = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(decode(read.base64).byteLength).toBe(BRIDGE_AUDIO_RING_MAX_BYTES - 1) + expect(read.droppedBytes).toBe(64) + // Cleared by the read that reported it: two reads must never count the same dropped byte. + const next = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(next.droppedBytes).toBe(0) + }) + + it('never holds more than the ring however many chunks arrive', async () => { + const { engine, emit } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + for (let index = 0; index < 400; index += 1) { + emit(pcm(1_024, index)) + } + const read = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(decode(read.base64).byteLength).toBeLessThanOrEqual(BRIDGE_AUDIO_RING_MAX_BYTES) + expect(decode(read.base64).byteLength + read.droppedBytes).toBe(400 * 1_024) + }) + + it('carries an interruption on the next read and stops reporting it after', async () => { + const { engine, emit, interrupt } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + emit(pcm(256, 9)) + interrupt('began') + const read = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(read.interruption).toBe('began') + // The capture is gone, but the bytes it produced are still the page's to drain. + expect(read.recording).toBe(false) + expect(decode(read.base64).byteLength).toBe(256) + const next = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(next.interruption).toBeNull() + }) + + it('keeps a capture the OS handed back, and ends the two it took away', async () => { + const { engine, interrupt } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + interrupt('ended') + const kept = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + // The kind still crosses — the page is told what happened — but the capture is still live, so + // the ring goes on filling and the page goes on draining it. + expect(kept.interruption).toBe('ended') + expect(kept.recording).toBe(true) + interrupt('blocked') + const lost = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(lost.recording).toBe(false) + }) + + it('refuses a read once the page has stopped', async () => { + const { engine } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + await expect(capture.serve('native.audio.stop', {})).resolves.toEqual({ stopped: true }) + await expect(capture.serve('native.audio.read', { maxBytes: 1_024 })).rejects.toSatisfy( + (error: unknown) => + error instanceof BridgeNativeVerbRefusedError && error.code === 'native_audio_not_capturing' + ) + // A second stop is the state the page already has, not a fault. + await expect(capture.serve('native.audio.stop', {})).resolves.toEqual({ stopped: false }) + }) + + it('refuses a read before any start', async () => { + const { engine } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await expect(capture.serve('native.audio.read', { maxBytes: 1_024 })).rejects.toSatisfy( + (error: unknown) => + error instanceof BridgeNativeVerbRefusedError && error.code === 'native_audio_not_capturing' + ) + }) + + it('takes the microphone off the moment a capture ends', async () => { + const { engine, emit, log } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + await capture.serve('native.audio.stop', {}) + expect(log).toContain('end') + expect(log).toContain('microphone-off') + // A late event from an engine that has not finished shutting down reaches nothing. + emit(pcm(1_024, 1)) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + const read = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(read.base64).toBe('') + }) + + it('replaces a capture a page left behind rather than refusing the new one', async () => { + // The page is a document that can navigate, fault or be swiped away mid-capture, and the shell + // is the only side that can notice. A second start therefore ends the first. + const { engine, emit, log } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + emit(pcm(2_048, 6)) + await expect(capture.serve('native.audio.start', { sampleRate: 16_000 })).resolves.toEqual({ + started: true, + sampleRate: 16_000, + permission: 'granted' + }) + expect(log.filter((entry) => entry === 'end')).toHaveLength(1) + const read = audioReadResultSchema.parse( + await capture.serve('native.audio.read', { maxBytes: BRIDGE_AUDIO_RING_MAX_BYTES }) + ) + expect(read.base64).toBe('') + }) + + it('leaves one capture behind when two starts race the permission prompt', async () => { + // A page can be reloaded while the OS prompt is up — the shell's own reason for replacing a + // capture rather than refusing one — and both starts then reach `listen()`. The second + // overwriting the first left the first's handlers subscribed for the app's lifetime, so the + // engine kept filling a ring nobody could read and `dispose` freed one of two. + const prompt: { release: () => void } = { release: () => {} } + const gate = new Promise((resolve) => { + prompt.release = resolve + }) + const { engine, log, liveListeners } = createTestEngine({ + permission: async () => { + await gate + return 'granted' + } + }) + const capture = createNativeAudioCapture(engine) + const first = capture.serve('native.audio.start', { sampleRate: 16_000 }) + const second = capture.serve('native.audio.start', { sampleRate: 16_000 }) + prompt.release() + await expect(first).resolves.toMatchObject({ started: true }) + await expect(second).resolves.toMatchObject({ started: true }) + expect(liveListeners()).toEqual({ microphone: 1, interruptions: 1 }) + capture.dispose() + expect(liveListeners()).toEqual({ microphone: 0, interruptions: 0 }) + // Two captures were opened and both were ended: one by the replacement, one by the dispose. + expect(log.filter((entry) => entry === 'end')).toHaveLength(2) + }) + + it('does not open a capture for a start that lands after the session ended', async () => { + const prompt: { release: () => void } = { release: () => {} } + const gate = new Promise((resolve) => { + prompt.release = resolve + }) + const { engine, liveListeners, log } = createTestEngine({ + permission: async () => { + await gate + return 'granted' + } + }) + const capture = createNativeAudioCapture(engine) + const pending = capture.serve('native.audio.start', { sampleRate: 16_000 }) + capture.dispose() + prompt.release() + await expect(pending).resolves.toMatchObject({ started: false }) + expect(liveListeners()).toEqual({ microphone: 0, interruptions: 0 }) + // And the device is left torn down. The open succeeded — on a phone that is `initialize()` + // bringing the audio session up — so a start that simply returned here would leave it up with + // nothing holding it: the local end is a no-op with no capture, and nobody else will call one. + expect(log).toContain('end') + }) + + it('tears the device down for a start that lost the race after opening', async () => { + const prompt: { release: () => void } = { release: () => {} } + const gate = new Promise((resolve) => { + prompt.release = resolve + }) + // The race lost after the permission, inside the open itself, which is the longer of the two. + const { engine, log, liveListeners } = createTestEngine({ + open: async (sampleRate) => { + await gate + return { opened: true, sampleRate } + } + }) + const capture = createNativeAudioCapture(engine) + const pending = capture.serve('native.audio.start', { sampleRate: 16_000 }) + capture.dispose() + prompt.release() + await expect(pending).resolves.toEqual({ + started: false, + sampleRate: 16_000, + permission: 'granted' + }) + expect(log.filter((entry) => entry === 'end')).toHaveLength(1) + expect(liveListeners()).toEqual({ microphone: 0, interruptions: 0 }) + }) + + it('ends a capture a stop asked for while its start was still opening', async () => { + const prompt: { release: () => void } = { release: () => {} } + const gate = new Promise((resolve) => { + prompt.release = resolve + }) + const { engine, liveListeners } = createTestEngine({ + permission: async () => { + await gate + return 'granted' + } + }) + const capture = createNativeAudioCapture(engine) + const started = capture.serve('native.audio.start', { sampleRate: 16_000 }) + const stopped = capture.serve('native.audio.stop', {}) + prompt.release() + await started + // The stop runs after the start it followed, so it ends the capture that start opened rather + // than finding nothing and leaving a live microphone behind it. + await expect(stopped).resolves.toEqual({ stopped: true }) + expect(liveListeners()).toEqual({ microphone: 0, interruptions: 0 }) + }) + + it('ends the capture when the page session does', async () => { + const { engine, log } = createTestEngine() + const capture = createNativeAudioCapture(engine) + await capture.serve('native.audio.start', { sampleRate: 16_000 }) + capture.dispose() + expect(log).toContain('end') + await expect(capture.serve('native.audio.read', { maxBytes: 16 })).rejects.toBeInstanceOf( + BridgeNativeVerbRefusedError + ) + }) +}) + +describe('the wake lock', () => { + it('holds a tag, answers what the device did, and gives it back', async () => { + const held: string[] = [] + const { serve } = createNativeWakelockServer({ + activate: async (tag) => { + held.push(`+${tag}`) + }, + deactivate: async (tag) => { + held.push(`-${tag}`) + } + }) + await expect(serve({ active: true, tag: 'orca-a' })).resolves.toEqual({ active: true }) + await expect(serve({ active: false, tag: 'orca-a' })).resolves.toEqual({ active: false }) + expect(held).toEqual(['+orca-a', '-orca-a']) + }) + + it('does not ask the device to drop a tag it never took', async () => { + const held: string[] = [] + const { serve } = createNativeWakelockServer({ + activate: async (tag) => { + held.push(`+${tag}`) + }, + deactivate: async (tag) => { + held.push(`-${tag}`) + } + }) + await expect(serve({ active: false, tag: 'orca-b' })).resolves.toEqual({ active: false }) + expect(held).toEqual([]) + }) + + it('reports a tag the device refused as not held', async () => { + const { serve } = createNativeWakelockServer({ + activate: async () => { + throw new Error('no keep-awake on this device') + }, + deactivate: async () => undefined + }) + await expect(serve({ active: true, tag: 'orca-c' })).rejects.toBeInstanceOf(Error) + }) + + it('keeps a tag recorded when the device refused to drop it, so a retry reaches the device', async () => { + // The page's owner queues a failed deactivation and retries it (`pendingCleanupTags` in + // `mobile-dictation-keep-awake.ts`). That retry arrives here as another `active: false`, and it + // has to reach the device: a shell that had already forgotten the tag answers "not held" + // without calling anything, and the native tag stays on for the life of the app. + const calls: string[] = [] + let refuse = true + const { serve } = createNativeWakelockServer({ + activate: async (tag) => { + calls.push(`+${tag}`) + }, + deactivate: async (tag) => { + calls.push(`-${tag}`) + if (refuse) { + throw new Error('the device would not drop the tag') + } + } + }) + await serve({ active: true, tag: 'orca-f' }) + // The refusal crosses, so the page's owner knows to queue a retry rather than believing it. + await expect(serve({ active: false, tag: 'orca-f' })).rejects.toBeInstanceOf(Error) + refuse = false + await expect(serve({ active: false, tag: 'orca-f' })).resolves.toEqual({ active: false }) + expect(calls).toEqual(['+orca-f', '-orca-f', '-orca-f']) + // And once it is really gone, a third release asks the device nothing. + await expect(serve({ active: false, tag: 'orca-f' })).resolves.toEqual({ active: false }) + expect(calls).toEqual(['+orca-f', '-orca-f', '-orca-f']) + }) + + it('keeps a tag a dispose could not drop, rather than forgetting it', async () => { + const calls: string[] = [] + const { serve, dispose } = createNativeWakelockServer({ + activate: async (tag) => { + calls.push(`+${tag}`) + }, + deactivate: async (tag) => { + calls.push(`-${tag}`) + throw new Error('the device would not drop the tag') + } + }) + await serve({ active: true, tag: 'orca-g' }) + dispose() + await Promise.resolve() + await Promise.resolve() + // Still recorded, so the owner's retry is still able to reach the device through this server. + await expect(serve({ active: false, tag: 'orca-g' })).rejects.toBeInstanceOf(Error) + expect(calls).toEqual(['+orca-g', '-orca-g', '-orca-g']) + }) + + it('gives back a tag whose activation landed after the session ended', async () => { + // The page is a document that can be swiped away mid-dictation, so a dispose can fall between + // the activate call and its reply. A tag recorded after that dispose is held by nobody and + // keeps the screen awake for the app's lifetime. + const held: string[] = [] + const gate: { release: () => void } = { release: () => {} } + const activated = new Promise((resolve) => { + gate.release = resolve + }) + const { serve, dispose } = createNativeWakelockServer({ + activate: async (tag) => { + await activated + held.push(`+${tag}`) + }, + deactivate: async (tag) => { + held.push(`-${tag}`) + } + }) + const pending = serve({ active: true, tag: 'orca-late' }) + dispose() + gate.release() + // Answered as not held, because by the time the device had it nobody wanted it. + await expect(pending).resolves.toEqual({ active: false }) + await Promise.resolve() + expect(held).toEqual(['+orca-late', '-orca-late']) + }) + + it('gives back every tag it still holds when the session ends', async () => { + const held: string[] = [] + const { serve, dispose } = createNativeWakelockServer({ + activate: async (tag) => { + held.push(`+${tag}`) + }, + deactivate: async (tag) => { + held.push(`-${tag}`) + } + }) + await serve({ active: true, tag: 'orca-d' }) + await serve({ active: true, tag: 'orca-e' }) + await serve({ active: false, tag: 'orca-d' }) + dispose() + await Promise.resolve() + // Only what was still held: a tag the page already gave back is not deactivated twice. + expect(held).toEqual(['+orca-d', '+orca-e', '-orca-d', '-orca-e']) + // And nothing is held afterwards, so a second dispose asks the device nothing. + dispose() + await Promise.resolve() + expect(held).toEqual(['+orca-d', '+orca-e', '-orca-d', '-orca-e']) + }) +}) diff --git a/mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.ts b/mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.ts new file mode 100644 index 00000000000..015e36f11f3 --- /dev/null +++ b/mobile/src/mobile-web-shell/bridge/bridge-audio-verbs.ts @@ -0,0 +1,162 @@ +import { z } from 'zod' +import { MOBILE_DICTATION_MAX_PENDING_AUDIO_BYTES } from '../../hooks/mobile-dictation-pending-audio-budget' + +/** + * The wire shapes of `native.audio.start`, `native.audio.read`, `native.audio.stop` and + * `native.wakelock.set`. + * + * Dictation is the page's, and the microphone is the shell's. The page holds the state machine the + * composer renders and speaks `speech.dictation.*` to the desktop, so the only thing that has to + * cross is the capability: raw PCM, and the wake tag that keeps the screen alive while it is + * captured. That is why audio is pulled rather than pushed. The `request`/`reply` table is the only + * page-facing seam the shell has, the one shell-to-page push there is belongs to an RPC + * `subscribe`, and a push lane for bytes the page immediately hands back would be a new frame kind + * for no gain. + * + * So the shell rings what the microphone produces and the page drains it. The ring is exactly the + * page's own pending-audio budget: the page already refuses to hold more unsent audio than that, + * so a second, different bound would be a second answer to the same question. + * + * The verbs are `audio.start|read|stop`, never `audio.readChunk`: a manifest grant name is held to + * `native(?:\.[a-z][a-z0-9]*){2,}` by `GRANT_NAME_PATTERN`, and `bundled-mobile-web-bundle.ts` + * parses the manifest whole, so one camel-cased segment is a bundle the phone refuses entire. + */ + +/** + * What the shell holds for a page that has not drained, in raw PCM bytes. + * + * Derived from the page's budget rather than written down beside it: `use-mobile-dictation.ts` + * already refuses to hold more than this in unsent audio and fails the dictation with + * `MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE` when it would. Two numbers that must agree are + * one that drifts, and a ring larger than the budget would hand the page bytes it will throw away. + */ +export const BRIDGE_AUDIO_RING_MAX_BYTES = MOBILE_DICTATION_MAX_PENDING_AUDIO_BYTES + +/** Whole base64 groups of the ring: four characters encode three bytes, so a full drain is exactly + * this many characters and never one more. A drain of the whole ring is 213,336 characters, which + * is a third of `BRIDGE_MAX_MESSAGE_BYTES` — so the largest read there can be still fits a frame. */ +export const BRIDGE_AUDIO_READ_MAX_BASE64_CHARS = Math.ceil(BRIDGE_AUDIO_RING_MAX_BYTES / 3) * 4 + +/** + * The rates a capture may run at. + * + * Wide rather than pinned to the 16 kHz the desktop transcribes at, because the device is what + * decides: an engine that will not open at the asked-for rate answers the rate it opened at, and a + * page that pinned one value here could not carry that answer back. + */ +export const BRIDGE_AUDIO_MIN_SAMPLE_RATE = 8_000 +export const BRIDGE_AUDIO_MAX_SAMPLE_RATE = 48_000 + +/** + * What the OS said about the microphone, as data rather than as a rejection. + * + * A denied microphone is the answer to `start`, not a fault in it: the page renders "permission + * denied" as a state the user can act on, and a rejection would reach the same screen as a shell + * that could not be talked to at all. `undetermined` is the arm that cannot be reached through a + * shell that prompts — the prompt is the shell's, inside `start`, as ruling 6b settled for `pick` — + * and is kept because an OS that answers neither still has to be describable. + */ +export const BRIDGE_AUDIO_PERMISSIONS = ['granted', 'denied', 'undetermined'] as const + +export type BridgeAudioPermission = (typeof BRIDGE_AUDIO_PERMISSIONS)[number] + +/** + * What the OS did to a capture that was running, in the vocabulary the native engines already emit. + * + * It rides the `read` reply rather than a fifth verb or a push: the page is already asking every + * 500 ms, so the longest an interruption can go unseen is that interval, and a page that heard it + * reaches exactly the state `onAudioInterruption` reaches natively. + */ +export const BRIDGE_AUDIO_INTERRUPTIONS = ['began', 'ended', 'blocked'] as const + +export type BridgeAudioInterruption = (typeof BRIDGE_AUDIO_INTERRUPTIONS)[number] + +/** + * Whether an interruption is the OS taking the capture away, rather than handing it back. + * + * `began` and `blocked` end it; `ended` on its own does not — that is the OS returning the session + * after, say, a notification chime, and a dictation that cancelled on it would end itself the + * moment the chime finished. One predicate because three places decide it: the shell, which stops + * filling its ring; the native seam, which raises it off `onAudioInterruption`; and the page, which + * raises it off a `read` reply. Two of them had drifted apart. + * + * Takes a string rather than the union, because the native engines hand over whatever they emit and + * a kind this build has no name for is not an interruption it can describe. + */ +export function bridgeAudioInterruptionEndsCapture(kind: string): boolean { + return kind === 'began' || kind === 'blocked' +} + +/** The longest wake tag the shell will hold. The dictation tag is the owner id and the dictation id + * joined, both minted from a clock and a random suffix, so this is roughly twice the longest one + * this build can produce and short enough that a page cannot park text in the shell's tag set. */ +export const BRIDGE_WAKELOCK_TAG_MAX_CHARS = 160 + +const BASE64_PATTERN = /^[A-Za-z0-9+/]*={0,2}$/ + +const sampleRateSchema = z + .number() + .int() + .min(BRIDGE_AUDIO_MIN_SAMPLE_RATE) + .max(BRIDGE_AUDIO_MAX_SAMPLE_RATE) + +// Strict, not stripping, for the reason every other verb's params are: the page and the shell are +// separate builds, and a param the shell silently drops is the shape of a verb that changed under +// a page that thought it had asked for something. +export const audioStartParamsSchema = z.strictObject({ sampleRate: sampleRateSchema }) + +/** + * `started` and `permission` both, because they answer different questions. + * + * A denied microphone is `started: false, permission: 'denied'`; an engine that would not open on a + * granted microphone is `started: false, permission: 'granted'`. A page that had only the boolean + * would send the user to Settings for a device fault, and one that had only the permission would + * sit in `recording` with no microphone behind it. + */ +export const audioStartResultSchema = z.strictObject({ + started: z.boolean(), + sampleRate: sampleRateSchema, + permission: z.enum(BRIDGE_AUDIO_PERMISSIONS) +}) + +export const audioReadParamsSchema = z.strictObject({ + maxBytes: z.number().int().min(1).max(BRIDGE_AUDIO_RING_MAX_BYTES) +}) + +/** + * One drain of the ring: the bytes, what the ring could not hold, and whether it is still filling. + * + * `droppedBytes` is what the page acts on rather than a diagnostic. The ring is the page's own + * budget, so a drop means the page is not draining as fast as the microphone fills — which is the + * `MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE` state the composer already renders. Counted + * since the previous read and cleared by it, so two reads never report the same dropped byte twice. + * + * `recording` is the shell's own state and not an echo of the page's: a capture the OS took away + * answers `false` with the last bytes still in the ring, so a page drains what it has and then + * stops rather than reading an empty ring forever. + */ +export const audioReadResultSchema = z.strictObject({ + base64: z.string().max(BRIDGE_AUDIO_READ_MAX_BASE64_CHARS).regex(BASE64_PATTERN), + droppedBytes: z.number().int().nonnegative(), + recording: z.boolean(), + interruption: z.enum(BRIDGE_AUDIO_INTERRUPTIONS).nullable() +}) + +export type BridgeAudioChunk = z.infer + +/** No params: there is one capture per page session, so there is nothing to name. */ +export const audioStopParamsSchema = z.strictObject({}) + +/** False for a session that was not capturing, which is not a fault: a page that stops twice, or + * stops after an interruption already ended the capture, asked for the state it already has. */ +export const audioStopResultSchema = z.strictObject({ stopped: z.boolean() }) + +export const wakelockSetParamsSchema = z.strictObject({ + active: z.boolean(), + tag: z.string().min(1).max(BRIDGE_WAKELOCK_TAG_MAX_CHARS) +}) + +/** Whether the tag is held after the call, which is what was asked for unless the device refused. + * Answered rather than assumed so the page's tag bookkeeping tracks the device and not its own + * intent — the same thing `activateKeepAwakeAsync` resolving tells the native owner. */ +export const wakelockSetResultSchema = z.strictObject({ active: z.boolean() }) diff --git a/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.test.ts b/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.test.ts index a4e5856baba..54070832f31 100644 --- a/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.test.ts +++ b/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.test.ts @@ -105,7 +105,11 @@ describe('the media verbs on the same seam', () => { 'native.clipboard.read', 'native.media.pick', 'native.media.read', - 'native.media.release' + 'native.media.release', + 'native.audio.start', + 'native.audio.read', + 'native.audio.stop', + 'native.wakelock.set' ]) }) diff --git a/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.ts b/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.ts index 7e67dc03a80..654055b779f 100644 --- a/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.ts +++ b/mobile/src/mobile-web-shell/bridge/bridge-native-verbs.ts @@ -1,4 +1,14 @@ import { z } from 'zod' +import { + audioReadParamsSchema, + audioReadResultSchema, + audioStartParamsSchema, + audioStartResultSchema, + audioStopParamsSchema, + audioStopResultSchema, + wakelockSetParamsSchema, + wakelockSetResultSchema +} from './bridge-audio-verbs' import { mediaPickParamsSchema, mediaPickResultSchema, @@ -31,7 +41,11 @@ export const BRIDGE_NATIVE_VERB_NAMES = [ 'native.clipboard.read', 'native.media.pick', 'native.media.read', - 'native.media.release' + 'native.media.release', + 'native.audio.start', + 'native.audio.read', + 'native.audio.stop', + 'native.wakelock.set' ] as const export type BridgeNativeVerb = (typeof BRIDGE_NATIVE_VERB_NAMES)[number] @@ -99,6 +113,25 @@ export const BRIDGE_NATIVE_VERBS: Readonly { expect(caught instanceof NativeVerbError && caught.reason).toBe('unreported') }) }) + +describe('what the surface offers a screen', () => { + it('carries no capability flag with nothing behind it', async () => { + const pair = createFakeBridgePortPair({}) + const verbs = await mount(pair) + // Pinned so an unread flag has to be added here on purpose. `canCaptureAudio` was one: four + // grants read and answered to nobody, where the fence that actually holds is the per-verb + // `ungranted` check every member already makes before a frame is sent. A flag no screen reads + // is a capability negotiation that exists only in this file. + expect(Object.keys(verbs).sort()).toEqual([ + 'canPickMedia', + 'canReadClipboardText', + 'canWriteClipboardText', + 'granted', + 'pickMedia', + 'readAudio', + 'readClipboardText', + 'readMedia', + 'releaseMedia', + 'setWakelock', + 'startAudio', + 'stopAudio', + 'writeClipboardText' + ]) + }) +}) diff --git a/mobile/src/mobile-web-shell/bridge/use-native-verbs.ts b/mobile/src/mobile-web-shell/bridge/use-native-verbs.ts index 4622b0e2856..5bb508f3e98 100644 --- a/mobile/src/mobile-web-shell/bridge/use-native-verbs.ts +++ b/mobile/src/mobile-web-shell/bridge/use-native-verbs.ts @@ -9,6 +9,13 @@ import { type BridgeMediaItem, type BridgeMediaSource } from './bridge-media-verbs' +import { + audioReadResultSchema, + audioStartResultSchema, + audioStopResultSchema, + wakelockSetResultSchema, + type BridgeAudioChunk +} from './bridge-audio-verbs' import { clipboardReadResultSchema, clipboardWriteResultSchema, @@ -66,6 +73,17 @@ export type NativeVerbs = { readMedia: (handle: string, offset: number, length: number) => Promise /** False for a handle this session no longer holds, which is not a fault. */ releaseMedia: (handle: string) => Promise + /** Opens the microphone, running the OS prompt if there is one. A denied microphone and an + * engine that would not open are both answers here rather than rejections. */ + startAudio: (sampleRate: number) => Promise> + /** One drain of the shell's ring. `maxBytes` above the ring is refused by the shell's schema, so + * a caller bounds its own ask rather than discovering the bound as a rejection. */ + readAudio: (maxBytes: number) => Promise + /** False for a session that was not capturing, which is not a fault. */ + stopAudio: () => Promise + /** Whether the tag is held after the call. The shell asks the device nothing for a tag it never + * took, so releasing one twice is not a fault either. */ + setWakelock: (active: boolean, tag: string) => Promise } /** @@ -120,6 +138,7 @@ export const NATIVE_VERB_REASONS = [ 'native_media_too_large', 'native_media_permission_denied', 'native_verb_not_a_stream', + 'native_audio_not_capturing', 'native_verb_not_a_verb', 'bridge_cap_exceeded', 'bridge_host_disposed', @@ -200,7 +219,13 @@ export function useNativeVerbs(): NativeVerbs { readMedia: (handle, offset, length) => call('native.media.read', { handle, offset, length }, mediaReadResultSchema), releaseMedia: async (handle) => - (await call('native.media.release', { handle }, mediaReleaseResultSchema)).released + (await call('native.media.release', { handle }, mediaReleaseResultSchema)).released, + startAudio: (sampleRate) => + call('native.audio.start', { sampleRate }, audioStartResultSchema), + readAudio: (maxBytes) => call('native.audio.read', { maxBytes }, audioReadResultSchema), + stopAudio: async () => (await call('native.audio.stop', {}, audioStopResultSchema)).stopped, + setWakelock: async (active, tag) => + (await call('native.wakelock.set', { active, tag }, wakelockSetResultSchema)).active } }, [client]) } diff --git a/mobile/src/mobile-web-shell/page-route-policy.test.ts b/mobile/src/mobile-web-shell/page-route-policy.test.ts index 00dfbb35899..3b39f18f9f1 100644 --- a/mobile/src/mobile-web-shell/page-route-policy.test.ts +++ b/mobile/src/mobile-web-shell/page-route-policy.test.ts @@ -95,7 +95,11 @@ describe('the grants this app implements', () => { 'native.clipboard.read', 'native.media.pick', 'native.media.read', - 'native.media.release' + 'native.media.release', + 'native.audio.start', + 'native.audio.read', + 'native.audio.stop', + 'native.wakelock.set' ]) }) diff --git a/mobile/src/platform/dictation-capture-bridge-budget.test.tsx b/mobile/src/platform/dictation-capture-bridge-budget.test.tsx new file mode 100644 index 00000000000..23c583c9bee --- /dev/null +++ b/mobile/src/platform/dictation-capture-bridge-budget.test.tsx @@ -0,0 +1,277 @@ +/** + * What one dictation costs the bridge, measured against the cap that actually bounds it. + * + * Not the frame cap: a chunk of 500 ms of 16 kHz PCM is 16,000 bytes, which is 21,336 characters of + * base64 against `BRIDGE_MAX_MESSAGE_BYTES` of 655,360. The bound is + * `BRIDGE_MAX_PENDING_REQUESTS`, because every `speech.dictation.chunk` is a forwarded request + * holding a slot for the whole desktop round trip. One request per native microphone event puts + * Android's 31.25 events a second against a two-second link at 62 of 64 slots — so the page refuses + * at its own call site before the desktop is even slow. + * + * Driven through the real port pair with the real shell handler and a desktop that answers after + * the link's own delay, so what this counts is frames that were really sent and replies that had + * really not arrived. + */ +import type { ReactElement } from 'react' +import { act, create } from 'react-test-renderer' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +// The provider module re-exports the screen hooks, and reaching the real ones imports the Expo +// runtime this test does not have. Nothing below calls one. +vi.mock('../transport/host-client-hooks', () => ({ + useDisconnectHostClient: () => () => {}, + useForceReconnect: () => () => Promise.resolve(), + useForgetHostClient: () => () => {}, + useHostClient: () => ({ client: null, clientId: null, state: 'disconnected' }), + usePrimeHosts: () => () => {}, + useRefreshHostClient: () => () => {} +})) + +import { RpcClientProvider } from '../transport/client-context.web' +import { BRIDGE_MAX_PENDING_REQUESTS } from '../mobile-web-shell/bridge/bridge-caps' +import { useNativeVerbs, type NativeVerbs } from '../mobile-web-shell/bridge/use-native-verbs' +import { + createFakeBridgePortPair, + type BridgePortPair +} from '../mobile-web-shell/bridge/bridge-port-pair-test-harness' +import { MobileDictationPendingAudioBudget } from '../hooks/mobile-dictation-pending-audio-budget' +import { enqueueMobileDictationAudioChunk } from '../hooks/mobile-dictation-audio-chunk' +import { createNativeAudioCapture, type NativeAudioEngine } from './native-audio' +import { createPageDictationCapture } from './dictation-capture.web' +import { + DICTATION_CAPTURE_DRAIN_INTERVAL_MS, + DICTATION_NATIVE_EVENT_INTERVAL_MS +} from './dictation-capture-contract' +import type { BridgeNativeVerb } from '../mobile-web-shell/bridge/bridge-native-verbs' + +/** The link the design prices against: a phone on a slow network to a desktop that answers. */ +const DESKTOP_ROUND_TRIP_MS = 2_000 + +/** One dictation long enough to reach a steady state at either rate. */ +const SESSION_MS = 10_000 + +/** 1,024 bytes of 16 kHz 16-bit PCM, which is what both native engines emit per event. */ +const NATIVE_EVENT_BYTES = 1_024 + +function pcm(byteLength: number): Uint8Array { + const bytes = new Uint8Array(byteLength) + for (let index = 0; index < byteLength; index += 1) { + bytes[index] = (index * 31 + 7) % 251 + } + return bytes +} + +/** A shell holding a real ring over an engine this test speaks into. */ +function createAudioShell() { + let microphone: ((bytes: Uint8Array) => void) | null = null + const engine: NativeAudioEngine = { + requestPermission: async () => 'granted', + open: async (sampleRate) => ({ opened: true, sampleRate }), + begin: () => true, + end: () => {}, + onMicrophoneData: (handler) => { + microphone = handler + return { + remove: () => { + microphone = null + } + } + }, + onInterruption: () => ({ remove: () => {} }) + } + const capture = createNativeAudioCapture(engine) + return { + speak: (bytes: Uint8Array) => microphone?.(bytes), + serveNativeVerb: (verb: BridgeNativeVerb, params: unknown): Promise => + capture.serve(verb, params) + } +} + +/** + * One dictation of `SESSION_MS`, drained at `drainIntervalMs`, over a link of + * `DESKTOP_ROUND_TRIP_MS`. + * + * The page's half is the two things that decide the cost: the drain, and the chunk each drain + * forwards. Sampled every native event, so the peak is the real one and not the value at the end. + */ +async function runDictation( + pair: BridgePortPair, + shell: ReturnType, + drainIntervalMs: number +): Promise<{ + peakInFlight: number + chunkRequests: number + refusals: number + freeSlots: number + framesToShell: number + framesToPage: number +}> { + const capture = createPageDictationCapture(await mountVerbs(pair), drainIntervalMs) + const pendingChunks = new Set>() + const pendingAudioBudget = new MobileDictationPendingAudioBudget() + const refusals: string[] = [] + let peakInFlight = 0 + let settled = 0 + + capture.onChunk((chunk) => { + if (chunk.droppedBytes > 0) { + refusals.push('dropped') + return + } + enqueueMobileDictationAudioChunk(pair.client, 'dictation-1', chunk, { + pendingChunks, + pendingAudioBudget, + shouldReleaseBudget: () => true, + failActiveDictation: (_id, error) => { + refusals.push(error instanceof Error ? error.message : String(error)) + } + }) + }) + + await capture.open() + capture.begin() + const framesBefore = { toShell: pair.toShell.length, toPage: pair.toPage.length } + + /** The desktop, answering each forwarded chunk one round trip after it arrived. */ + function answerDesktop(): void { + for (; settled < pair.rpc.requests.length; settled += 1) { + const request = pair.rpc.requests[settled] + if (request === undefined) { + return + } + setTimeout(() => request.resolve({ id: 'x', ok: true, result: {} }), DESKTOP_ROUND_TRIP_MS) + } + } + + const steps = Math.floor(SESSION_MS / DICTATION_NATIVE_EVENT_INTERVAL_MS) + for (let step = 0; step < steps; step += 1) { + shell.speak(pcm(NATIVE_EVENT_BYTES)) + await vi.advanceTimersByTimeAsync(DICTATION_NATIVE_EVENT_INTERVAL_MS) + await pair.flush() + answerDesktop() + peakInFlight = Math.max(peakInFlight, pendingChunks.size) + } + const framesToShell = pair.toShell.length - framesBefore.toShell + const framesToPage = pair.toPage.length - framesBefore.toPage + const freeSlots = await probeFreeSlots(pair) + capture.end() + await pair.flush() + return { + peakInFlight, + freeSlots, + framesToShell, + framesToPage, + chunkRequests: pair.rpc.requests.filter( + (request) => request.method === 'speech.dictation.chunk' + ).length, + refusals: refusals.length + } +} + +/** + * How much of the in-flight window is left for everything else the page does, at this moment. + * + * Ordinary requests are fired until the shell refuses one over the cap, which is the only honest + * measure of a window: a count of what dictation holds says nothing about what is free unless the + * cap is the thing that answers. + */ +async function probeFreeSlots(pair: BridgePortPair): Promise { + let free = 0 + for (let attempt = 0; attempt <= BRIDGE_MAX_PENDING_REQUESTS; attempt += 1) { + let refused = false + const pending = pair.client.sendRequest('worktree.list').catch(() => { + refused = true + }) + await pair.flush() + // A request the shell accepted is one the fake desktop is holding, which is a slot spent. + if (refused) { + await pending + return free + } + free += 1 + } + return free +} + +const held: { verbs: NativeVerbs | null } = { verbs: null } + +function Screen(): null { + held.verbs = useNativeVerbs() + return null +} + +function render(pair: BridgePortPair): ReactElement { + return ( + + + + ) +} + +/** The page's own verb surface, taken off a mounted screen exactly as a composer takes it. */ +async function mountVerbs(pair: BridgePortPair): Promise { + await pair.flush() + act(() => { + create(render(pair)) + }) + const verbs = held.verbs + if (verbs === null) { + throw new Error('nothing mounted') + } + return verbs +} + +beforeEach(() => { + held.verbs = null + vi.useFakeTimers() +}) + +afterEach(() => { + vi.useRealTimers() +}) + +describe('what one dictation spends of the bridge', () => { + it('fills the in-flight window at one request per native microphone event', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + await pair.flush() + const spent = await runDictation(pair, shell, DICTATION_NATIVE_EVENT_INTERVAL_MS) + // 31.25 events a second against a two-second link is 62.5 requests outstanding, which is the + // whole window. Dictation does not overflow it on its own — it leaves nothing for anything + // else, which is the same defect one screen later. + expect(spent.peakInFlight).toBeGreaterThanOrEqual(BRIDGE_MAX_PENDING_REQUESTS - 2) + expect(spent.freeSlots).toBeLessThanOrEqual(2) + }, 60_000) + + it('spends four slots of sixty-four on the shipped drain', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + await pair.flush() + const spent = await runDictation(pair, shell, DICTATION_CAPTURE_DRAIN_INTERVAL_MS) + // Two chunks a second over a two-second link: four outstanding, and the same 42 KiB/s. + expect(spent.peakInFlight).toBeLessThanOrEqual(6) + expect(spent.refusals).toBe(0) + // The window is still the page's: everything else a screen does still fits. + expect(spent.freeSlots).toBeGreaterThanOrEqual(BRIDGE_MAX_PENDING_REQUESTS - 8) + // What ten seconds of dictation costs the bridge, measured: 38 frames out — 19 reads and 19 + // chunk forwards — and 34 back, the four missing being the chunk replies the desktop is still + // holding at the count. Under four frames a second, each under 4% of the cap, on a transport + // that moves them inside one process. + expect({ toShell: spent.framesToShell, toPage: spent.framesToPage }).toEqual({ + toShell: 38, + toPage: 34 + }) + // Ten seconds at two a second, give or take the drain that lands on the boundary. + expect(spent.chunkRequests).toBeGreaterThanOrEqual( + Math.floor(SESSION_MS / DICTATION_CAPTURE_DRAIN_INTERVAL_MS) - 1 + ) + expect(spent.chunkRequests).toBeLessThanOrEqual( + Math.ceil(SESSION_MS / DICTATION_CAPTURE_DRAIN_INTERVAL_MS) + 1 + ) + }, 60_000) + + it('ships the batched drain, not the native event rate', () => { + expect(DICTATION_CAPTURE_DRAIN_INTERVAL_MS).toBe(500) + expect(DICTATION_CAPTURE_DRAIN_INTERVAL_MS).toBeGreaterThan(DICTATION_NATIVE_EVENT_INTERVAL_MS) + }) +}) diff --git a/mobile/src/platform/dictation-capture-compile-fence.ts b/mobile/src/platform/dictation-capture-compile-fence.ts new file mode 100644 index 00000000000..c992c05eb37 --- /dev/null +++ b/mobile/src/platform/dictation-capture-compile-fence.ts @@ -0,0 +1,35 @@ +import type { MicrophoneDataEvent } from '@orca/expo-two-way-audio' +import { enqueueMobileDictationAudioChunk } from '../hooks/mobile-dictation-audio-chunk' +import type { DictationCaptureChunk } from './dictation-capture-contract' +import type { RpcClient } from '../transport/rpc-client' + +// Why this file exists: the chunk sender takes what the capture seam hands over, and a microphone +// event is not that — it carries no drop count, and the page's whole reason for a drop count is +// audio the shell's ring could not hold. Typed as the event, the sender accepted either and read +// `droppedBytes` off neither, so a page that had dropped audio sent it as though nothing was +// missing. Every expect-error below is that claim as an assertion: tsc fails on a directive that +// stops catching an error, so `tsc -p tsconfig.json` is the gate. Nothing here runs and no app code +// imports it. + +declare const client: RpcClient +declare const dictationId: string +declare const queue: Parameters[3] +declare const bytes: Uint8Array + +const _fenceChunkIsAccepted: void = enqueueMobileDictationAudioChunk( + client, + dictationId, + { data: bytes, droppedBytes: 0 }, + queue +) + +const _fenceEventIsRefused: void = enqueueMobileDictationAudioChunk( + client, + dictationId, + // @ts-expect-error a microphone event is not a capture chunk: it says nothing about dropped audio + { data: bytes } satisfies MicrophoneDataEvent, + queue +) + +// @ts-expect-error the seam normalises the bytes, so a chunk never carries a raw buffer +const _fenceBufferIsRefused: DictationCaptureChunk = { data: new ArrayBuffer(8), droppedBytes: 0 } diff --git a/mobile/src/platform/dictation-capture-contract.ts b/mobile/src/platform/dictation-capture-contract.ts new file mode 100644 index 00000000000..2ea31080c57 --- /dev/null +++ b/mobile/src/platform/dictation-capture-contract.ts @@ -0,0 +1,109 @@ +/** + * Where dictation's audio comes from, as the hook that drives it sees it. + * + * One seam, two hosts. Natively it is `@orca/expo-two-way-audio` and `expo-keep-awake` called + * directly; on the page it is `native.audio.start|read|stop` and `native.wakelock.set` over the + * bridge. Everything above it — the five composer states, the generation guards, the pending-audio + * budget, where a transcript is routed — is the same code on both, because the part that differs + * is the capability and the part that does not is the product. + * + * The shape is the native one: a permission and an open, a start and a stop, two event lanes and a + * wake tag. That is deliberate. The page's pull is what `dictation-capture.web.ts` turns into these + * events, so the flow above the seam cannot tell which host it is on, and the native half is the + * calls it always made in the order it always made them. + */ + +/** + * One piece of captured audio. + * + * `data` is the field a microphone event already has, so a chunk is what + * `enqueueMobileDictationAudioChunk` has always taken and the sender is untouched by the seam: raw + * PCM is the one form both hosts agree on, the budget counts it, and the base64 for the wire is + * built after the reserve exactly as it was. The page pays a decode for that — the shell's reply + * carries base64 — which at 32 KB/s is the price of one chunk shape rather than two. + */ +export type DictationCaptureChunk = { + readonly data: Uint8Array + /** + * Audio the capture had and could not hand over. + * + * Always zero natively, where the microphone reaches this process directly. On the page it is the + * shell's ring filling faster than the drain empties it, which is the same condition the budget + * refuses on — so both reach `MOBILE_DICTATION_CONNECTION_SLOW_ERROR_MESSAGE`, which is a state + * the composer already renders. + */ + readonly droppedBytes: number +} + +/** Why a capture would not open. Both are device answers rather than faults: a shell that refused + * the call at all rejects instead, with the reason on it. */ +export type DictationCaptureOpen = + | { readonly ok: true } + | { readonly ok: false; readonly reason: 'permission-denied' | 'unavailable' } + +export type DictationCaptureSubscription = { readonly remove: () => void } + +/** The two calls that keep the screen alive while a dictation runs, and nothing else: the tag + * bookkeeping, its retries and its timeouts are host-independent and stay above this. */ +export type DictationKeepAwakeDevice = { + readonly activate: (tag: string) => Promise + readonly deactivate: (tag: string) => Promise +} + +export type DictationCapture = { + /** Runs the OS permission prompt if there is one and brings the engine up. */ + readonly open: () => Promise + /** Starts producing chunks. False is a device that would not, which rolls the start back. */ + readonly begin: () => boolean + /** + * Stops producing chunks, after handing over everything the capture still holds. + * + * Asynchronous because of the page, where the audio lives in the shell's ring and the last one + * of them has to be fetched: up to one drain interval of the utterance's tail is sitting there + * when the user lifts the button, and no timer is coming for it. Natively that audio already + * reached the hook as it was produced, so there the promise is already resolved. + * + * Never rejects. It runs on every exit including a throw, where a rejection would replace what + * brought us here with a complaint about cleaning up after it. + */ + readonly end: () => Promise + /** Gives the capture up for good; the screen's unmount calls it. */ + readonly release: () => void + readonly onChunk: ( + handler: (chunk: DictationCaptureChunk) => void + ) => DictationCaptureSubscription + /** + * The capture was taken away — a call, another app, a shell that no longer has one. + * + * No argument, because what the flow does about any of them is the same: cancel, release the + * tag, tell the desktop. Natively this is `onAudioInterruption`'s `began` and `blocked`; on the + * page it is the same two riding a `read` reply, plus a read the shell refused, which is a + * capture that is gone by another name. + */ + readonly onInterruption: (handler: () => void) => DictationCaptureSubscription + readonly keepAwake: DictationKeepAwakeDevice +} + +/** + * The rate a native engine produces microphone events at. + * + * 1,024 bytes of 16 kHz 16-bit PCM is 32 ms, so both engines emit 31.25 times a second. Named here + * because the page's drain is priced against it: a page that read once per native event would put + * 63 of the bridge's 64 in-flight slots into dictation on a two-second link. + */ +export const DICTATION_NATIVE_EVENT_INTERVAL_MS = 32 + +/** + * How often the page drains the shell's ring, and therefore how often it forwards a chunk. + * + * Priced against `BRIDGE_MAX_PENDING_REQUESTS`, which is what bounds dictation rather than the + * frame cap. Every `speech.dictation.chunk` is a forwarded request holding one of 64 slots for a + * whole desktop round trip, so at the native event rate a two-second link leaves 62 of them in + * flight and nothing for the screen around it. At this interval it is four, on the same 42 KiB/s: + * the batch costs nothing but latency, and half a second of it is below what a transcript that + * arrives after `finish` can show. + * + * The frame is never the bound. Half a second of 16 kHz 16-bit PCM is 16,000 bytes, which is + * 21,336 characters of base64 against a 655,360-byte frame cap — 3.3% of one. + */ +export const DICTATION_CAPTURE_DRAIN_INTERVAL_MS = 500 diff --git a/mobile/src/platform/dictation-capture.test.ts b/mobile/src/platform/dictation-capture.test.ts new file mode 100644 index 00000000000..84410d4c571 --- /dev/null +++ b/mobile/src/platform/dictation-capture.test.ts @@ -0,0 +1,152 @@ +/** + * The native half of the capture seam: the calls it makes, and which interruptions end a capture. + * + * Thin by design — this file is five device calls behind a shape the page can answer — but the + * interruption rule is shared with `dictation-capture.web.ts` and is exactly where the two drifted: + * the page treated every interruption as a loss while this one has always gated on `began` and + * `blocked`, so an `ended` on its own cancelled a live dictation on the page and nothing natively. + */ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const engine = vi.hoisted(() => ({ + listeners: new Map void>(), + calls: new Array(), + /** Set by a case that wants the JSI binding to fail the way a device can. */ + throwOn: new Set() +})) + +vi.mock('@orca/expo-two-way-audio', () => ({ + addExpoTwoWayAudioEventListener: (name: string, handler: (event: { data: unknown }) => void) => { + engine.listeners.set(name, handler) + return { + remove: () => { + engine.listeners.delete(name) + } + } + }, + initialize: () => { + engine.calls.push('initialize') + return Promise.resolve(true) + }, + requestMicrophonePermissionsAsync: () => { + engine.calls.push('permission') + return Promise.resolve({ granted: true }) + }, + tearDown: () => { + engine.calls.push('tearDown') + if (engine.throwOn.has('tearDown')) { + throw new Error('the audio session would not tear down') + } + }, + toggleRecording: (on: boolean) => { + engine.calls.push(`toggleRecording(${String(on)})`) + if (engine.throwOn.has(`toggleRecording(${String(on)})`)) { + throw new Error('the audio engine would not stop') + } + return true + } +})) +vi.mock('expo-keep-awake', () => ({ + activateKeepAwakeAsync: (tag: string) => { + engine.calls.push(`+${tag}`) + return Promise.resolve() + }, + deactivateKeepAwake: (tag: string) => { + engine.calls.push(`-${tag}`) + return Promise.resolve() + } +})) + +import { useDictationCapture } from './dictation-capture' + +beforeEach(() => { + engine.listeners.clear() + engine.calls.length = 0 + engine.throwOn.clear() +}) + +describe('which interruptions end a native capture', () => { + it('ends on the two the OS means by it, and not on the one it does not', () => { + const capture = useDictationCapture() + let interrupted = 0 + capture.onInterruption(() => { + interrupted += 1 + }) + const fire = (data: string) => engine.listeners.get('onAudioInterruption')?.({ data }) + // `ended` on its own is the OS handing the session back, not taking it away. A dictation that + // cancelled on it would end itself the moment a notification chime finished playing. + fire('ended') + expect(interrupted).toBe(0) + fire('began') + expect(interrupted).toBe(1) + fire('blocked') + expect(interrupted).toBe(2) + // And a kind from a newer engine is not an interruption this build can describe. + fire('something-new') + expect(interrupted).toBe(2) + }) +}) + +describe('the calls the native half makes', () => { + it('asks for the permission, opens the engine, and reports what it got', async () => { + const capture = useDictationCapture() + await expect(capture.open()).resolves.toEqual({ ok: true }) + expect(engine.calls).toEqual(['permission', 'initialize']) + expect(capture.begin()).toBe(true) + await capture.end() + capture.release() + expect(engine.calls).toEqual([ + 'permission', + 'initialize', + 'toggleRecording(true)', + 'toggleRecording(false)', + 'tearDown' + ]) + }) + + it('hands every microphone event over with nothing dropped', () => { + const capture = useDictationCapture() + const chunks: { data: Uint8Array; droppedBytes: number }[] = [] + capture.onChunk((chunk) => chunks.push(chunk)) + const bytes = Uint8Array.from([1, 2, 3, 4]) + engine.listeners.get('onMicrophoneData')?.({ data: bytes }) + expect(chunks).toEqual([{ data: bytes, droppedBytes: 0 }]) + }) + + it('takes the wake tag through expo-keep-awake', async () => { + const capture = useDictationCapture() + await capture.keepAwake.activate('orca-a') + await capture.keepAwake.deactivate('orca-a') + expect(engine.calls).toEqual(['+orca-a', '-orca-a']) + }) +}) + +describe('a device whose audio session will not shut down', () => { + it('resolves `end` rather than rejecting it, which the contract promises', async () => { + // `end` is async, so a throw from the binding becomes a rejection. Every caller reaches it as + // `void capture.end()` inside a synchronous try/catch, which cannot see a rejection — so the + // failure left the app with an unhandled rejection instead of a logged one, and the cleanup + // that was meant to keep going was never the thing at risk. + engine.throwOn.add('toggleRecording(false)') + const capture = useDictationCapture() + await expect(capture.end()).resolves.toBeUndefined() + expect(engine.calls).toEqual(['toggleRecording(false)']) + }) + + it('does not throw out of `release`, which runs bare in the unmount path', async () => { + engine.throwOn.add('tearDown') + const capture = useDictationCapture() + expect(() => capture.release()).not.toThrow() + await Promise.resolve() + expect(engine.calls).toEqual(['tearDown']) + }) + + it('still stops the engine when the tear-down is the half that fails', async () => { + engine.throwOn.add('tearDown') + const capture = useDictationCapture() + await capture.end() + capture.release() + await Promise.resolve() + expect(engine.calls).toEqual(['toggleRecording(false)', 'tearDown']) + }) +}) diff --git a/mobile/src/platform/dictation-capture.ts b/mobile/src/platform/dictation-capture.ts new file mode 100644 index 00000000000..656272b59a9 --- /dev/null +++ b/mobile/src/platform/dictation-capture.ts @@ -0,0 +1,75 @@ +import { + addExpoTwoWayAudioEventListener, + initialize, + requestMicrophonePermissionsAsync, + tearDown, + toggleRecording +} from '@orca/expo-two-way-audio' +import { activateKeepAwakeAsync, deactivateKeepAwake } from 'expo-keep-awake' +import { bridgeAudioInterruptionEndsCapture } from '../mobile-web-shell/bridge/bridge-audio-verbs' +import type { DictationCapture } from './dictation-capture-contract' + +/** + * The device's microphone, which is where dictation's audio has always come from. + * + * Every call is the one the hook used to make, in the order it made it, because this half is the + * seam's shape rather than a translation of it: a permission and an open, a start and a stop, the + * two event lanes `@orca/expo-two-way-audio` emits, and the wake tag. What moved is where they are + * written, so the page can answer the same shape without the flow above knowing which it holds. + */ +const nativeDictationCapture: DictationCapture = { + open: async () => { + const permission = await requestMicrophonePermissionsAsync() + if (!permission.granted) { + return { ok: false, reason: 'permission-denied' } + } + return (await initialize()) ? { ok: true } : { ok: false, reason: 'unavailable' } + }, + begin: () => toggleRecording(true), + /** + * Already resolved: every microphone event reached the hook as the engine produced it, so there + * is nothing held back for a stop to hand over. + * + * And it never rejects, which the contract promises because of how it is called: every site + * reaches it as `void capture.end()` inside a synchronous `try`, which cannot see a rejection. A + * binding that threw left an unhandled rejection rather than a logged failure, so the throw is + * swallowed here where there is somewhere to log it. + */ + end: async () => { + try { + toggleRecording(false) + } catch (error) { + console.error('Failed to stop microphone recording', error) + } + }, + /** Same reason, and a sharper one: this runs bare in the unmount path, where a throw would take + * the rest of the cleanup — the wake tag and the desktop's cancel — with it. */ + release: () => { + try { + tearDown() + } catch (error) { + console.error('Failed to tear down the audio session', error) + } + }, + onChunk: (handler) => + addExpoTwoWayAudioEventListener('onMicrophoneData', (event) => { + const raw = event.data + handler({ + data: raw instanceof Uint8Array ? raw : new Uint8Array(raw), + // Nothing is ever dropped on the way here: this process is where the microphone is, and + // what the flow cannot keep up with is the pending-audio budget's to refuse, not this. + droppedBytes: 0 + }) + }), + onInterruption: (handler) => + addExpoTwoWayAudioEventListener('onAudioInterruption', (event) => { + if (bridgeAudioInterruptionEndsCapture(event.data)) { + handler() + } + }), + keepAwake: { activate: activateKeepAwakeAsync, deactivate: deactivateKeepAwake } +} + +export function useDictationCapture(): DictationCapture { + return nativeDictationCapture +} diff --git a/mobile/src/platform/dictation-capture.web.test.tsx b/mobile/src/platform/dictation-capture.web.test.tsx new file mode 100644 index 00000000000..27f6d22aeed --- /dev/null +++ b/mobile/src/platform/dictation-capture.web.test.tsx @@ -0,0 +1,515 @@ +/** + * The page's form of the capture seam: the shell holds the microphone and the page drains it. + * + * Driven through the real port pair against the real shell handler, so what this reads is the four + * verbs leaving the page and the chunks the drain builds out of what came back — the same path the + * composer's mic button takes. Every refusal is a case, because the whole point of the seam is + * that a shell saying no reaches the screen as the state it is rather than as a crash. + */ +import type { ReactElement } from 'react' +import { act, create } from 'react-test-renderer' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +// The provider module re-exports the screen hooks, and reaching the real ones imports the Expo +// runtime this test does not have. Nothing below calls one. +vi.mock('../transport/host-client-hooks', () => ({ + useDisconnectHostClient: () => () => {}, + useForceReconnect: () => () => Promise.resolve(), + useForgetHostClient: () => () => {}, + useHostClient: () => ({ client: null, clientId: null, state: 'disconnected' }), + usePrimeHosts: () => () => {}, + useRefreshHostClient: () => () => {} +})) + +import { RpcClientProvider } from '../transport/client-context.web' +import { BRIDGE_AUDIO_RING_MAX_BYTES } from '../mobile-web-shell/bridge/bridge-audio-verbs' +import { BridgeNativeVerbRefusedError } from '../mobile-web-shell/bridge-host-errors' +import { + createFakeBridgePortPair, + type BridgePortPair +} from '../mobile-web-shell/bridge/bridge-port-pair-test-harness' +import { NativeVerbError } from '../mobile-web-shell/bridge/use-native-verbs' +import { MOBILE_DICTATION_PCM_SAMPLE_RATE } from '../hooks/mobile-dictation-pending-audio-budget' +import { createNativeAudioCapture, type NativeAudioEngine } from './native-audio' +import { createNativeWakelockServer } from './native-wakelock' +import { useDictationCapture } from './dictation-capture.web' +import { DICTATION_CAPTURE_DRAIN_INTERVAL_MS } from './dictation-capture-contract' +import type { BridgeNativeVerb } from '../mobile-web-shell/bridge/bridge-native-verbs' +import type { DictationCapture, DictationCaptureChunk } from './dictation-capture-contract' + +/** The four verbs, served by the real shell handlers over an engine a case drives. */ +function createAudioShell( + options: { + permission?: 'granted' | 'denied' | 'undetermined' + opens?: boolean + refuse?: (verb: BridgeNativeVerb) => Error | null + } = {} +) { + let microphone: ((bytes: Uint8Array) => void) | null = null + let interrupt: ((kind: 'began' | 'ended' | 'blocked') => void) | null = null + const engine: NativeAudioEngine = { + requestPermission: async () => options.permission ?? 'granted', + open: async (sampleRate) => ({ opened: options.opens !== false, sampleRate }), + begin: () => true, + end: () => {}, + onMicrophoneData: (handler) => { + microphone = handler + return { + remove: () => { + microphone = null + } + } + }, + onInterruption: (handler) => { + interrupt = handler + return { + remove: () => { + interrupt = null + } + } + } + } + const capture = createNativeAudioCapture(engine) + const { serve: wakelock } = createNativeWakelockServer({ + activate: async () => undefined, + deactivate: async () => undefined + }) + const calls: string[] = [] + return { + calls, + speak: (bytes: Uint8Array) => microphone?.(bytes), + interrupt: (kind: 'began' | 'ended' | 'blocked') => interrupt?.(kind), + serveNativeVerb: (verb: BridgeNativeVerb, params: unknown): Promise => { + calls.push(verb) + const refusal = options.refuse?.(verb) ?? null + if (refusal !== null) { + return Promise.reject(refusal) + } + return verb === 'native.wakelock.set' ? wakelock(params) : capture.serve(verb, params) + } + } +} + +function pcm(byteLength: number, seed = 1): Uint8Array { + const bytes = new Uint8Array(byteLength) + for (let index = 0; index < byteLength; index += 1) { + bytes[index] = (index * 31 + seed) % 251 + } + return bytes +} + +const held: { capture: DictationCapture | null } = { capture: null } + +function Screen(): null { + held.capture = useDictationCapture() + return null +} + +function render(pair: BridgePortPair): ReactElement { + return ( + + + + ) +} + +async function mount(pair: BridgePortPair): Promise { + await pair.flush() + act(() => { + create(render(pair)) + }) + const capture = held.capture + if (capture === null) { + throw new Error('nothing mounted') + } + return capture +} + +/** One drain interval of fake time, plus the microtasks the read and its reply ride. */ +async function tick(pair: BridgePortPair, intervals = 1): Promise { + for (let index = 0; index < intervals; index += 1) { + await act(async () => { + await vi.advanceTimersByTimeAsync(DICTATION_CAPTURE_DRAIN_INTERVAL_MS) + await pair.flush() + }) + } +} + +beforeEach(() => { + held.capture = null + vi.useFakeTimers() +}) + +afterEach(() => { + vi.useRealTimers() +}) + +describe('opening a capture on the page', () => { + it('asks the shell to start and reports the rate it opened at', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await expect(capture.open()).resolves.toEqual({ ok: true }) + expect(shell.calls).toEqual(['native.audio.start']) + }) + + it('reports a denied microphone as the state it is, never as a rejection', async () => { + const shell = createAudioShell({ permission: 'denied' }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await expect(capture.open()).resolves.toEqual({ ok: false, reason: 'permission-denied' }) + }) + + it('reports an engine that would not open apart from a permission', async () => { + const shell = createAudioShell({ opens: false }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await expect(capture.open()).resolves.toEqual({ ok: false, reason: 'unavailable' }) + }) + + it('rejects as a native verb error when the route was never granted the audio verbs', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ + serveNativeVerb: shell.serveNativeVerb, + routeGrants: ['navigate', 'storage'] + }) + const capture = await mount(pair) + await expect(capture.open()).rejects.toSatisfy( + (error: unknown) => error instanceof NativeVerbError && error.reason === 'ungranted' + ) + // Refused before a frame is sent: an ungranted verb costs no in-flight slot. + expect(shell.calls).toEqual([]) + }) + + it('rejects as a native verb error when the shell refuses the start', async () => { + const shell = createAudioShell({ + refuse: (verb) => + verb === 'native.audio.start' + ? new BridgeNativeVerbRefusedError('native_verb_failed', 'no microphone on this device') + : null + }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await expect(capture.open()).rejects.toSatisfy( + (error: unknown) => error instanceof NativeVerbError && error.reason === 'native_verb_failed' + ) + }) +}) + +describe('draining the shell ring', () => { + it('delivers what the microphone produced, encoded once, with nothing dropped', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + const chunks: DictationCaptureChunk[] = [] + const sub = capture.onChunk((chunk) => chunks.push(chunk)) + await capture.open() + expect(capture.begin()).toBe(true) + shell.speak(pcm(1_024)) + await tick(pair) + expect(chunks).toHaveLength(1) + expect(chunks[0]?.droppedBytes).toBe(0) + // The bytes the microphone produced, in order and unaltered by the crossing. + expect(Array.from(chunks[0]?.data ?? [])).toEqual(Array.from(pcm(1_024))) + sub.remove() + }) + + it('delivers nothing for a silent interval rather than an empty chunk', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + const chunks: DictationCaptureChunk[] = [] + capture.onChunk((chunk) => chunks.push(chunk)) + await capture.open() + capture.begin() + await tick(pair, 3) + expect(chunks).toEqual([]) + // A budget the page never spends on silence: the reads happened, the chunks did not. + expect(shell.calls.filter((verb) => verb === 'native.audio.read').length).toBeGreaterThan(0) + }) + + it('carries what the shell ring could not hold', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + const chunks: DictationCaptureChunk[] = [] + capture.onChunk((chunk) => chunks.push(chunk)) + await capture.open() + capture.begin() + shell.speak(pcm(BRIDGE_AUDIO_RING_MAX_BYTES)) + shell.speak(pcm(2_048)) + await tick(pair) + expect(chunks[0]?.droppedBytes).toBe(2_048) + }) + + it('delivers the tail still in the ring before it stops the shell', async () => { + // The utterance's last 400 ms sits in the shell's ring when the user lifts the button: less + // than one drain interval, so no timer will ever come for it. Natively that audio is already + // in the hook's hands by the time recording stops, so a page that dropped it would transcribe + // a sentence with its ending cut off. + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + const chunks: DictationCaptureChunk[] = [] + capture.onChunk((chunk) => chunks.push(chunk)) + await capture.open() + capture.begin() + await tick(pair) + // 400 ms of 16 kHz 16-bit PCM, spoken after the last drain and before the next one. + const tail = pcm(12_288, 11) + shell.speak(tail) + await act(async () => { + await vi.advanceTimersByTimeAsync(400) + await pair.flush() + }) + expect(chunks).toEqual([]) + await act(async () => { + await capture.end() + await pair.flush() + }) + expect(chunks).toHaveLength(1) + expect(Array.from(chunks[0]?.data ?? [])).toEqual(Array.from(tail)) + }) + + it('stops the shell after the last read, never before it', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await capture.open() + capture.begin() + shell.speak(pcm(2_048, 12)) + await act(async () => { + await capture.end() + await pair.flush() + }) + // A stop that landed first would have taken the capture away, and the read would be refused. + expect(shell.calls.slice(-2)).toEqual(['native.audio.read', 'native.audio.stop']) + }) + + it('reads nothing once the capture has ended', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await capture.open() + capture.begin() + await tick(pair) + const before = shell.calls.length + await act(async () => { + await capture.end() + await pair.flush() + }) + const afterEnd = shell.calls.length + await tick(pair, 3) + // The last read, the stop, and then nothing: a timer left running would keep asking a shell + // that no longer has a capture. + expect(shell.calls.slice(before, afterEnd)).toEqual(['native.audio.read', 'native.audio.stop']) + expect(shell.calls.slice(afterEnd)).toEqual([]) + }) +}) + +describe('a capture the page loses', () => { + it('reaches the interruption lane when the OS takes the microphone', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + let interrupted = 0 + capture.onInterruption(() => { + interrupted += 1 + }) + await capture.open() + capture.begin() + shell.interrupt('began') + await tick(pair) + expect(interrupted).toBe(1) + }) + + it('does not end a live capture for an interruption that only ended', async () => { + // The native seam gates on `began` and `blocked`; the page must gate on the same two, or a + // notification chime finishing cancels a dictation on the page and nothing natively. + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + let interrupted = 0 + capture.onInterruption(() => { + interrupted += 1 + }) + await capture.open() + capture.begin() + shell.interrupt('ended') + await tick(pair) + expect(interrupted).toBe(0) + // And the drain is still running, so the capture really did survive it. + const before = shell.calls.length + await tick(pair) + expect(shell.calls.length).toBeGreaterThan(before) + shell.interrupt('blocked') + await tick(pair) + expect(interrupted).toBe(1) + }) + + it('reaches the same lane when the shell refuses the read', async () => { + const shell = createAudioShell({ + refuse: (verb) => + verb === 'native.audio.read' + ? new BridgeNativeVerbRefusedError( + 'native_audio_not_capturing', + 'this session has no capture to read from' + ) + : null + }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + let interrupted = 0 + capture.onInterruption(() => { + interrupted += 1 + }) + await capture.open() + capture.begin() + await tick(pair, 3) + // Once, and then the drain stops: a shell with no capture will not grow one. + expect(interrupted).toBe(1) + }) + + it('does not loop when the interruption handler ends the capture, as the hook does', async () => { + // The hook's handler is `() => void cancel()`, and `cancel` reaches `capture.end()` + // synchronously through `closeDictationAudio`. So a refused read inside `end` re-enters `end`, + // whose own last read is refused too: without an idempotent `end` that recursion issues bridge + // reads until the page is torn down. + const shell = createAudioShell({ + refuse: (verb) => + verb === 'native.audio.read' + ? new BridgeNativeVerbRefusedError( + 'native_audio_not_capturing', + 'this session has no capture to read from' + ) + : null + }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + capture.onInterruption(() => { + void capture.end() + }) + await capture.open() + capture.begin() + await act(async () => { + await capture.end() + await pair.flush() + }) + await tick(pair, 3) + const reads = shell.calls.filter((verb) => verb === 'native.audio.read').length + expect(reads).toBeLessThanOrEqual(2) + }) + + it('reads again for the next dictation after an end, rather than staying ended', async () => { + // `end` is idempotent for the life of one capture, and the seam is memoised per client, so a + // second dictation on the same screen has to be able to drain. + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + const chunks: DictationCaptureChunk[] = [] + capture.onChunk((chunk) => chunks.push(chunk)) + await capture.open() + capture.begin() + await act(async () => { + await capture.end() + await pair.flush() + }) + await capture.open() + capture.begin() + shell.speak(pcm(512, 21)) + await tick(pair) + expect(Array.from(chunks.at(-1)?.data ?? [])).toEqual(Array.from(pcm(512, 21))) + }) + + it('ends a capture opened after the last end, though `begin` never ran between them', async () => { + // `open` starts the shell recording, and the hook sets `activeIdRef` before the desktop + // session exists: a start that goes stale after that point cleans up through `capture.end()` + // while `commitRecordingStart` -- the only caller of `begin` -- never runs. An `end` still + // holding the previous dictation's settled promise answers from it and stops nothing, leaving + // the shell recording a capture no page is draining. + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await capture.open() + capture.begin() + await act(async () => { + await capture.end() + await pair.flush() + }) + await capture.open() + const stopsBefore = shell.calls.filter((verb) => verb === 'native.audio.stop').length + await act(async () => { + await capture.end() + await pair.flush() + }) + const stopsAfter = shell.calls.filter((verb) => verb === 'native.audio.stop').length + expect(stopsAfter).toBe(stopsBefore + 1) + }) + + it('swallows a refused stop, because a capture that will not end is not the page to fix', async () => { + const shell = createAudioShell({ + refuse: (verb) => + verb === 'native.audio.stop' + ? new BridgeNativeVerbRefusedError('native_verb_failed', 'the engine would not stop') + : null + }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await capture.open() + capture.begin() + expect(() => capture.end()).not.toThrow() + expect(() => capture.release()).not.toThrow() + await pair.flush() + }) +}) + +describe('the wake tag on the page', () => { + it('takes and gives back a tag through the shell', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await expect(capture.keepAwake.activate('orca-mobile-dictation:1')).resolves.toBeUndefined() + await expect(capture.keepAwake.deactivate('orca-mobile-dictation:1')).resolves.toBeUndefined() + expect(shell.calls).toEqual(['native.wakelock.set', 'native.wakelock.set']) + }) + + it('rejects when the shell refuses the tag, so the owner can retry rather than believe it', async () => { + const shell = createAudioShell({ + refuse: (verb) => + verb === 'native.wakelock.set' + ? new BridgeNativeVerbRefusedError('native_verb_failed', 'no wake lock on this device') + : null + }) + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await expect(capture.keepAwake.activate('orca-a')).rejects.toBeInstanceOf(NativeVerbError) + }) + + it('rejects when the route was never granted the wake lock', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ + serveNativeVerb: shell.serveNativeVerb, + routeGrants: ['navigate', 'native.audio.start', 'native.audio.read', 'native.audio.stop'] + }) + const capture = await mount(pair) + await expect(capture.keepAwake.activate('orca-a')).rejects.toSatisfy( + (error: unknown) => error instanceof NativeVerbError && error.reason === 'ungranted' + ) + }) +}) + +describe('the rate the page asks for', () => { + it('is the one the desktop transcribes at', async () => { + const shell = createAudioShell() + const pair = createFakeBridgePortPair({ serveNativeVerb: shell.serveNativeVerb }) + const capture = await mount(pair) + await capture.open() + const started = pair + .readToShell() + .find((frame) => frame.type === 'request' && frame.method === 'native.audio.start') + expect(started).toBeDefined() + expect(started?.type === 'request' && started.params).toEqual({ + sampleRate: MOBILE_DICTATION_PCM_SAMPLE_RATE + }) + }) +}) diff --git a/mobile/src/platform/dictation-capture.web.ts b/mobile/src/platform/dictation-capture.web.ts new file mode 100644 index 00000000000..3eed2afa438 --- /dev/null +++ b/mobile/src/platform/dictation-capture.web.ts @@ -0,0 +1,250 @@ +import { useMemo } from 'react' +import { + BRIDGE_AUDIO_RING_MAX_BYTES, + bridgeAudioInterruptionEndsCapture, + type BridgeAudioChunk +} from '../mobile-web-shell/bridge/bridge-audio-verbs' +import { useNativeVerbs, type NativeVerbs } from '../mobile-web-shell/bridge/use-native-verbs' +import { MOBILE_DICTATION_PCM_SAMPLE_RATE } from '../hooks/mobile-dictation-pending-audio-budget' +import { + DICTATION_CAPTURE_DRAIN_INTERVAL_MS, + type DictationCapture, + type DictationCaptureChunk, + type DictationCaptureOpen, + type DictationCaptureSubscription +} from './dictation-capture-contract' + +/** + * Web sibling: the page has no microphone, so the shell holds one and the page drains it. + * + * Pulled rather than pushed, because the page-facing seam is request/reply by rule and the one + * shell-to-page push there is belongs to an RPC `subscribe`. A push lane for bytes the page hands + * straight back to the desktop would be a new frame kind, capability-negotiated, carrying audio + * that is already in this process. + * + * So the shell rings and this drains on a timer, turning replies into the events + * `dictation-capture.ts` gets from the engine directly. Above the seam neither host is visible: the + * same state machine, the same budget, the same routing. + * + * Every refusal rejects as the `NativeVerbError` the bridge built, with the reason on it — except + * a read, whose refusal is a capture that is gone and reaches the interruption lane instead, + * because that is the state it is and the one the flow already knows how to leave. + */ + +/** One read takes the whole ring: the shell bounds what it holds, so a smaller ask would leave + * audio behind for no gain and a larger one is refused by the verb's own schema. */ +const READ_MAX_BYTES = BRIDGE_AUDIO_RING_MAX_BYTES + +/** + * The shell's reply back into the bytes a chunk carries. + * + * The sender encodes them again on the way to the desktop, which is a decode and an encode of + * 32 KB a second inside one process — the price of one chunk shape rather than two, and of a + * pending-audio budget that counts the same raw bytes on both hosts. + */ +function decodeBase64(base64: string): Uint8Array { + const binary = atob(base64) + const bytes = new Uint8Array(binary.length) + for (let index = 0; index < binary.length; index += 1) { + bytes[index] = binary.charCodeAt(index) + } + return bytes +} + +type Handlers = Set + +function subscribe( + handlers: Handlers, + handler: Handler +): DictationCaptureSubscription { + handlers.add(handler) + return { + remove: () => { + handlers.delete(handler) + } + } +} + +/** Split from the hook so a caller can drive it with a client of its own; the hook is the wiring. */ +export function createPageDictationCapture( + verbs: NativeVerbs, + drainIntervalMs: number = DICTATION_CAPTURE_DRAIN_INTERVAL_MS +): DictationCapture { + const chunkHandlers: Handlers<(chunk: DictationCaptureChunk) => void> = new Set() + const interruptionHandlers: Handlers<() => void> = new Set() + let timer: ReturnType | null = null + /** The end in flight; null when none is running. Cleared on settle rather than held, because + * `begin` is not guaranteed to run between two ends and a finished one must shadow neither. */ + let ending: Promise | null = null + /** The screen went away: no later end reads or stops again. Cleared by `begin`. */ + let released = false + /** The read in flight, if there is one. At most one: two would double the slots dictation spends + * and can settle out of order, which is a splice of two moments reaching the transcriber as + * speech. Held rather than flagged so a stop can wait for it before taking its own turn. */ + let reading: Promise | null = null + + function stopDraining(): void { + if (timer !== null) { + clearInterval(timer) + timer = null + } + } + + function interrupted(): void { + // Before the handlers, so a handler that ends the capture finds the drain already stopped. + stopDraining() + for (const handler of interruptionHandlers) { + handler() + } + } + + function deliver(reply: BridgeAudioChunk): void { + if (reply.base64.length > 0 || reply.droppedBytes > 0) { + const chunk: DictationCaptureChunk = { + data: decodeBase64(reply.base64), + droppedBytes: reply.droppedBytes + } + for (const handler of chunkHandlers) { + handler(chunk) + } + } + // The same two kinds the native seam ends on: an `ended` on its own is the OS handing the + // session back and leaves a live capture alone. `recording` is the shell's own state and ends + // it whatever the kind — a capture it no longer has is gone however it went. + if ( + !reply.recording || + (reply.interruption !== null && bridgeAudioInterruptionEndsCapture(reply.interruption)) + ) { + interrupted() + } + } + + async function readOnce(): Promise { + try { + deliver(await verbs.readAudio(READ_MAX_BYTES)) + } catch { + // A read the shell refused is a capture it no longer has, whatever the code says. The flow + // above leaves the same way it leaves a phone call, which is the honest answer: there is no + // microphone, and there will not be one without another start. + interrupted() + } + } + + function drain(): Promise { + if (reading !== null) { + return reading + } + const run = readOnce().finally(() => { + reading = null + }) + reading = run + return run + } + + /** Best effort, and deliberately quiet, for the reason `end` never rejects. */ + async function stopShell(): Promise { + await verbs.stopAudio().catch(() => undefined) + } + + /** + * The tail, then the stop, in that order. + * + * Whatever is in the ring when the user lifts the button is up to one interval of what they + * actually said, and no timer is coming for it — `stopDraining` has just cancelled the one that + * was. Stopping first would take the capture away and the read after it would be refused, so the + * order here is the whole fix. An in-flight drain is awaited before the last read rather than + * raced with it, because two reads settling out of order splice two moments together. + */ + async function runEndCapture(): Promise { + stopDraining() + await reading + reading = null + await readOnce() + await stopShell() + } + + /** + * Idempotent while one end is in flight, which is what stops a refused read looping. + * + * The hook's interruption handler is `() => void cancel()`, and `cancel` reaches `end()` + * synchronously through `closeDictationAudio`. So the last read here can raise an interruption + * that calls straight back into this function, whose own last read is refused for the same + * reason — the shell has no capture — and the recursion issues bridge reads until the page runs + * out of memory. Returning the in-flight promise makes the re-entrant call a no-op rather than a + * second read; the recursion happens while that promise is still pending, so guarding the flight + * is enough and outliving it is not required. + * + * Cleared on settle, because a finished end must not answer for the next capture. `open` starts + * the shell recording and the hook can reach `end` before `begin` — a start that goes stale + * after `activeIdRef` is set cleans up that way, and `begin` is the only thing that would have + * cleared a latch — so an end held past its own flight would report a stop it never issued and + * leave the shell holding a live microphone. + */ + function endCapture(): Promise { + // A released capture has already stopped the shell and has nobody to hand a tail to. + if (released) { + return Promise.resolve() + } + if (ending !== null) { + return ending + } + // Assigned before anything can await, so a handler re-entering from inside the read below + // finds it set rather than starting a second end. + const run = runEndCapture().finally(() => { + // Only its own flight: `begin` may have started a newer capture while this one settled. + if (ending === run) { + ending = null + } + }) + ending = run + return run + } + + return { + open: async (): Promise => { + const started = await verbs.startAudio(MOBILE_DICTATION_PCM_SAMPLE_RATE) + if (started.started) { + return { ok: true } + } + return { + ok: false, + reason: started.permission === 'granted' ? 'unavailable' : 'permission-denied' + } + }, + begin: () => { + // The shell began capturing inside `start`; this is the page's half, which is the drain. + ending = null + released = false + stopDraining() + timer = setInterval(() => { + void drain() + }, drainIntervalMs) + return true + }, + end: endCapture, + // No last read: a release is the screen going away, and there is nobody left to hand the tail + // to. The shell sweeps the ring with the capture. + release: () => { + // Marked released so a later `end` neither reads nor stops again: the screen is going away. + // A flag rather than a settled `ending`, which now clears itself and would unlatch this. + released = true + stopDraining() + void stopShell() + }, + onChunk: (handler) => subscribe(chunkHandlers, handler), + onInterruption: (handler) => subscribe(interruptionHandlers, handler), + keepAwake: { + activate: async (tag) => { + await verbs.setWakelock(true, tag) + }, + deactivate: async (tag) => { + await verbs.setWakelock(false, tag) + } + } + } +} + +export function useDictationCapture(): DictationCapture { + const verbs = useNativeVerbs() + return useMemo(() => createPageDictationCapture(verbs), [verbs]) +} diff --git a/mobile/src/platform/native-audio-device.ts b/mobile/src/platform/native-audio-device.ts new file mode 100644 index 00000000000..49dd4a8ccf8 --- /dev/null +++ b/mobile/src/platform/native-audio-device.ts @@ -0,0 +1,72 @@ +import { + addExpoTwoWayAudioEventListener, + initialize, + requestMicrophonePermissionsAsync, + tearDown, + toggleRecording +} from '@orca/expo-two-way-audio' +import { activateKeepAwakeAsync, deactivateKeepAwake } from 'expo-keep-awake' +import { BRIDGE_AUDIO_INTERRUPTIONS } from '../mobile-web-shell/bridge/bridge-audio-verbs' +import type { NativeAudioEngine } from './native-audio' +import type { WakelockDevice } from './native-wakelock' + +/** + * The device calls the audio and wake-lock verbs actually make. + * + * Separated from the servers for the media device's reason: importing `@orca/expo-two-way-audio` + * reaches a JSI binding that only exists in a device build, so a module naming it cannot be driven + * in a unit test at all — and the platform facts worth naming are here rather than spread through + * the handler. + */ + +/** + * The rate the engine actually opened at. + * + * `initialize()` takes no rate and answers a boolean: both engines open at their own fixed rate, + * which is the 16 kHz the desktop transcribes at and the one `MOBILE_DICTATION_PCM_SAMPLE_RATE` + * already names. So the asked-for rate is honoured only when it is that one, and anything else is + * answered with what the device will really produce rather than accepted and then ignored. + */ +export const NATIVE_AUDIO_DEVICE_SAMPLE_RATE = 16_000 + +function readInterruption(data: string): (typeof BRIDGE_AUDIO_INTERRUPTIONS)[number] | null { + return BRIDGE_AUDIO_INTERRUPTIONS.find((kind) => kind === data) ?? null +} + +export const nativeAudioDeviceEngine: NativeAudioEngine = { + requestPermission: async () => { + const permission = await requestMicrophonePermissionsAsync() + // The prompt is the OS's, run inside `start`, and this is what it decided. `canAskAgain` on a + // refusal is the "ask again later" state, which the page cannot act on differently: either way + // it has no microphone now and shows the same screen. + return permission.granted ? 'granted' : permission.canAskAgain ? 'undetermined' : 'denied' + }, + open: async (sampleRate) => ({ + opened: sampleRate === NATIVE_AUDIO_DEVICE_SAMPLE_RATE && (await initialize()), + sampleRate: NATIVE_AUDIO_DEVICE_SAMPLE_RATE + }), + begin: () => toggleRecording(true), + end: () => { + toggleRecording(false) + tearDown() + }, + onMicrophoneData: (handler) => + addExpoTwoWayAudioEventListener('onMicrophoneData', (event) => { + const raw = event.data + handler(raw instanceof Uint8Array ? raw : new Uint8Array(raw)) + }), + onInterruption: (handler) => + addExpoTwoWayAudioEventListener('onAudioInterruption', (event) => { + // A kind this build has no name for is not reported: the page switches over the list, and a + // string from a newer engine would reach it as an interruption it cannot describe. + const kind = readInterruption(event.data) + if (kind !== null) { + handler(kind) + } + }) +} + +export const nativeWakelockDevice: WakelockDevice = { + activate: (tag) => activateKeepAwakeAsync(tag), + deactivate: (tag) => deactivateKeepAwake(tag) +} diff --git a/mobile/src/platform/native-audio.ts b/mobile/src/platform/native-audio.ts new file mode 100644 index 00000000000..3d885533988 --- /dev/null +++ b/mobile/src/platform/native-audio.ts @@ -0,0 +1,247 @@ +import { + BRIDGE_AUDIO_RING_MAX_BYTES, + bridgeAudioInterruptionEndsCapture, + audioReadParamsSchema, + audioStartParamsSchema, + audioStopParamsSchema, + type BridgeAudioInterruption, + type BridgeAudioPermission +} from '../mobile-web-shell/bridge/bridge-audio-verbs' +import type { BridgeNativeVerb } from '../mobile-web-shell/bridge/bridge-native-verbs' +import { BridgeNativeVerbRefusedError } from '../mobile-web-shell/bridge-host-errors' +import { bytesToBase64 } from '../hooks/mobile-dictation-session-state' + +/** + * The device side of `native.audio.start`, `read` and `stop`. + * + * The microphone opens here, inside the shell, which is the whole reason these are verbs: a page + * served from a custom scheme has no `getUserMedia` worth having, and the OS permission prompt is + * the shell's to run. What crosses back is PCM, because the page is what speaks + * `speech.dictation.*` to the desktop — transcription runs there, and putting that protocol in the + * binary would freeze it until a store release. + * + * Every engine call is injectable for the reason the media verbs' are: the arms worth pinning — a + * denied microphone, an engine that will not open, a ring that filled, an interruption — are the + * ones a simulator makes expensive, and none of them is a fact about Swift. + */ + +/** The audio engine as this handler needs it: a permission, an open, a run, and two event lanes. */ +export type NativeAudioEngine = { + /** Runs the OS prompt if the OS runs one, and answers what it decided. */ + readonly requestPermission: () => Promise + /** Brings the engine up and answers the rate it actually opened at. */ + readonly open: (sampleRate: number) => Promise<{ opened: boolean; sampleRate: number }> + /** Starts producing microphone events. False is a device that would not. */ + readonly begin: () => boolean + /** Stops producing them and releases the session. Called on every exit, including a throw. */ + readonly end: () => void + readonly onMicrophoneData: (handler: (bytes: Uint8Array) => void) => { remove: () => void } + readonly onInterruption: (handler: (kind: BridgeAudioInterruption) => void) => { + remove: () => void + } +} + +/** + * What the microphone produced and the page has not taken yet, bounded by the page's own budget. + * + * The newcomer is dropped rather than the oldest, which is the same decision + * `MobileDictationPendingAudioBudget.tryReserve` makes on the page: what a page does about a drop + * is fail the dictation, so recency buys nothing, and dropping from the front would hand the page + * a splice of two moments that reads as speech nobody said. + */ +class NativeAudioRing { + private readonly chunks: Uint8Array[] = [] + private pendingBytes = 0 + private droppedBytes = 0 + + append(bytes: Uint8Array): void { + if (bytes.byteLength === 0) { + return + } + if (this.pendingBytes + bytes.byteLength > BRIDGE_AUDIO_RING_MAX_BYTES) { + this.droppedBytes += bytes.byteLength + return + } + this.chunks.push(bytes) + this.pendingBytes += bytes.byteLength + } + + /** Up to `maxBytes` from the front, splitting the chunk the bound falls inside, plus everything + * the ring refused since the previous drain. Cleared by the drain that reports it. */ + drain(maxBytes: number): { bytes: Uint8Array; droppedBytes: number } { + const taking = Math.min(maxBytes, this.pendingBytes) + const out = new Uint8Array(taking) + let written = 0 + while (written < taking) { + const head = this.chunks[0] + if (head === undefined) { + break + } + const room = taking - written + if (head.byteLength <= room) { + out.set(head, written) + written += head.byteLength + this.chunks.shift() + continue + } + out.set(head.subarray(0, room), written) + written += room + this.chunks[0] = head.subarray(room) + } + this.pendingBytes -= written + const droppedBytes = this.droppedBytes + this.droppedBytes = 0 + return { bytes: out, droppedBytes } + } +} + +type Capture = { + readonly ring: NativeAudioRing + readonly stopListening: () => void + recording: boolean + /** The interruption not yet carried to the page. One slot, because what a page does about any of + * them is the same and a queue would report a stale one after the live one. */ + interruption: BridgeAudioInterruption | null +} + +export type NativeAudioCapture = { + readonly serve: (verb: BridgeNativeVerb, params: unknown) => Promise + /** Ends whatever is running. The page session's end and the screen's unmount both call it. */ + readonly dispose: () => void +} + +export function createNativeAudioCapture(engine: NativeAudioEngine): NativeAudioCapture { + let capture: Capture | null = null + let disposed = false + /** + * Starts and stops run one at a time, in the order the page asked for them. + * + * Both of them await the device, and both decide what `capture` is when they come back. Two + * starts overlapping — a page reloaded while the OS prompt is up, which is the very case the + * replacement rule below exists for — each reached `listen()` and the second overwrote the + * first's handlers without removing them, leaving the engine calling into a capture nobody could + * read for the life of the app. A stop overlapping a start found nothing to end and the start + * opened a microphone after it. + * + * Reads stay off this queue: they must not wait behind an opening capture, and a read with no + * capture is already a refusal rather than a guess. + */ + let queue: Promise = Promise.resolve() + + function enqueue(action: () => Promise): Promise { + // On both settle paths: a start that failed must not wedge every stop behind it. + const run = queue.then(action, action) + queue = run.then( + () => undefined, + () => undefined + ) + return run + } + + function end(): boolean { + if (capture === null) { + return false + } + capture.stopListening() + capture = null + engine.end() + return true + } + + function listen(): Capture { + const ring = new NativeAudioRing() + const microphone = engine.onMicrophoneData((bytes) => { + ring.append(bytes) + }) + const interruptions = engine.onInterruption((kind) => { + if (capture === null) { + return + } + capture.interruption = kind + // A capture the page has given up on must not go on filling the ring behind it. The rule is + // the seam's own, so the shell, the device and the page all end on the same two kinds. + if (bridgeAudioInterruptionEndsCapture(kind)) { + capture.recording = false + } + }) + return { + ring, + stopListening: () => { + microphone.remove() + interruptions.remove() + }, + recording: true, + interruption: null + } + } + + async function start(params: unknown): Promise { + const { sampleRate } = audioStartParamsSchema.parse(params) + // A page is a document that can navigate, fault or be swiped away mid-capture, and the shell is + // the only side that notices. So a second start replaces the first rather than refusing it, + // which would leave the microphone held by a document that is gone. + end() + const permission = await engine.requestPermission() + if (permission !== 'granted') { + return { started: false, sampleRate, permission } + } + const opened = await engine.open(sampleRate) + if (!opened.opened) { + return { started: false, sampleRate: opened.sampleRate, permission } + } + // The session can end while the device is still opening. Nothing subscribes after that: the + // dispose has already run, and a capture opened behind it would have no owner to stop it. + // The open succeeded though — on a phone that is `initialize()` bringing the audio session up — + // so it is torn down here. `end()` below is a no-op with no capture, and nobody else will call + // one, so simply returning would leave the device's session up for the life of the app. + if (disposed) { + engine.end() + return { started: false, sampleRate: opened.sampleRate, permission } + } + capture = listen() + if (!engine.begin()) { + end() + return { started: false, sampleRate: opened.sampleRate, permission } + } + return { started: true, sampleRate: opened.sampleRate, permission } + } + + function read(params: unknown): unknown { + const { maxBytes } = audioReadParamsSchema.parse(params) + const live = capture + if (live === null) { + throw new BridgeNativeVerbRefusedError( + 'native_audio_not_capturing', + 'this session has no capture to read from' + ) + } + const drained = live.ring.drain(maxBytes) + const interruption = live.interruption + live.interruption = null + return { + base64: bytesToBase64(drained.bytes), + droppedBytes: drained.droppedBytes, + recording: live.recording, + interruption + } + } + + return { + serve: async (verb, params) => { + if (verb === 'native.audio.start') { + return enqueue(() => start(params)) + } + if (verb === 'native.audio.read') { + return read(params) + } + audioStopParamsSchema.parse(params) + // Queued so a stop that followed a start ends the capture that start opened, rather than + // finding nothing and leaving a live microphone behind it. + return enqueue(async () => ({ stopped: end() })) + }, + dispose: () => { + disposed = true + end() + } + } +} diff --git a/mobile/src/platform/native-wakelock.test.ts b/mobile/src/platform/native-wakelock.test.ts new file mode 100644 index 00000000000..6ff03fc8139 --- /dev/null +++ b/mobile/src/platform/native-wakelock.test.ts @@ -0,0 +1,140 @@ +import { describe, expect, it } from 'vitest' +import { createNativeWakelockServer, type WakelockDevice } from './native-wakelock' + +/** A device whose every call is resolved by the case, so an interleaving can be built by hand. */ +function createGatedDevice(): { + device: WakelockDevice + calls: string[] + settle: (call: string) => void + refuse: (call: string, error: Error) => void +} { + const calls: string[] = [] + const gates = new Map void; reject: (error: Error) => void }>() + const gate = (call: string): Promise => { + calls.push(call) + return new Promise((resolve, reject) => { + gates.set(call, { resolve: () => resolve(), reject }) + }) + } + return { + calls, + device: { + activate: (tag) => gate(`activate:${tag}`), + deactivate: (tag) => gate(`deactivate:${tag}`) + }, + settle: (call) => { + const pending = gates.get(call) + if (!pending) { + throw new Error(`nothing is waiting on ${call}`) + } + gates.delete(call) + pending.resolve() + }, + refuse: (call, error) => { + const pending = gates.get(call) + if (!pending) { + throw new Error(`nothing is waiting on ${call}`) + } + gates.delete(call) + pending.reject(error) + } + } +} + +/** Lets the gated calls above reach the awaits inside the server. */ +const settleMicrotasks = (): Promise => new Promise((resolve) => setTimeout(resolve, 0)) + +describe('a tag that lands after the session ended', () => { + it('keeps recording a tag whose compensating release the device refused', async () => { + // The set means "the device still has this tag". After `activate` resolves the device has it, + // so a compensating `deactivate` the device refuses must leave the tag recorded: nothing else + // walks the set once `dispose` has run, and a tag recorded nowhere is a screen that stays + // awake for the life of the app. The refusal reaches the caller so its retry path can run. + const gated = createGatedDevice() + const server = createNativeWakelockServer(gated.device) + const served = server.serve({ active: true, tag: 'orca-mobile-dictation:1:a' }) + await settleMicrotasks() + server.dispose() + gated.settle('activate:orca-mobile-dictation:1:a') + await settleMicrotasks() + gated.refuse( + 'deactivate:orca-mobile-dictation:1:a', + new Error('the device would not give the tag back') + ) + await expect(served).rejects.toThrow('the device would not give the tag back') + + // The tag is still recorded, so a later release reaches the device rather than answering + // "not held" without calling anything. + const release = server.serve({ active: false, tag: 'orca-mobile-dictation:1:a' }) + await settleMicrotasks() + expect(gated.calls).toEqual([ + 'activate:orca-mobile-dictation:1:a', + 'deactivate:orca-mobile-dictation:1:a', + 'deactivate:orca-mobile-dictation:1:a' + ]) + gated.settle('deactivate:orca-mobile-dictation:1:a') + await expect(release).resolves.toEqual({ active: false }) + }) +}) + +describe('a release issued while its own activate is still in flight', () => { + it('leaves the device off, because the last request said off', async () => { + // Without a queue the release reads `held` before the activate has recorded anything, finds + // nothing, deactivates nothing and reports `active: false` — and then the activate lands and + // the device stays on, holding a tag the page has already said it does not want. + const gated = createGatedDevice() + const server = createNativeWakelockServer(gated.device) + const activated = server.serve({ active: true, tag: 'orca-mobile-dictation:1:b' }) + await settleMicrotasks() + const released = server.serve({ active: false, tag: 'orca-mobile-dictation:1:b' }) + await settleMicrotasks() + gated.settle('activate:orca-mobile-dictation:1:b') + await settleMicrotasks() + await expect(activated).resolves.toEqual({ active: true }) + gated.settle('deactivate:orca-mobile-dictation:1:b') + await expect(released).resolves.toEqual({ active: false }) + expect(gated.calls).toEqual([ + 'activate:orca-mobile-dictation:1:b', + 'deactivate:orca-mobile-dictation:1:b' + ]) + }) + + it('orders two tags independently, so one slow device call cannot hold up another', async () => { + // The precondition the case above needs: the queue is per tag, not one chain for the server. + const gated = createGatedDevice() + const server = createNativeWakelockServer(gated.device) + const first = server.serve({ active: true, tag: 'orca-mobile-dictation:1:c' }) + const second = server.serve({ active: true, tag: 'orca-mobile-dictation:1:d' }) + await settleMicrotasks() + expect(gated.calls).toEqual([ + 'activate:orca-mobile-dictation:1:c', + 'activate:orca-mobile-dictation:1:d' + ]) + gated.settle('activate:orca-mobile-dictation:1:d') + await expect(second).resolves.toEqual({ active: true }) + gated.settle('activate:orca-mobile-dictation:1:c') + await expect(first).resolves.toEqual({ active: true }) + }) + + it('does not ask the device again for a tag a queued release already gave back', async () => { + // `dispose` queues behind the tag's own operations, so by the time it runs the release ahead + // of it may have returned the tag. Deactivating an unheld tag is a native call this module + // does not make: its failure would read to the page as a lock it could not drop. + const gated = createGatedDevice() + const server = createNativeWakelockServer(gated.device) + const activated = server.serve({ active: true, tag: 'orca-mobile-dictation:1:e' }) + await settleMicrotasks() + gated.settle('activate:orca-mobile-dictation:1:e') + await expect(activated).resolves.toEqual({ active: true }) + const released = server.serve({ active: false, tag: 'orca-mobile-dictation:1:e' }) + await settleMicrotasks() + server.dispose() + gated.settle('deactivate:orca-mobile-dictation:1:e') + await expect(released).resolves.toEqual({ active: false }) + await settleMicrotasks() + expect(gated.calls).toEqual([ + 'activate:orca-mobile-dictation:1:e', + 'deactivate:orca-mobile-dictation:1:e' + ]) + }) +}) diff --git a/mobile/src/platform/native-wakelock.ts b/mobile/src/platform/native-wakelock.ts new file mode 100644 index 00000000000..0c785df0e67 --- /dev/null +++ b/mobile/src/platform/native-wakelock.ts @@ -0,0 +1,120 @@ +import { wakelockSetParamsSchema } from '../mobile-web-shell/bridge/bridge-audio-verbs' + +/** + * The device side of `native.wakelock.set`, on the shell where `expo-keep-awake` exists. + * + * Dictation holds the screen awake from the moment recording starts until the transcript is back, + * because a screen lock mid-processing suspends the app and loses it. On the page that tag has to + * be asked for, which is this verb; natively the same seam calls `expo-keep-awake` directly. + * + * The shell tracks what it is holding for two reasons. A page that releases a tag it never took + * asks the device nothing, because `deactivateKeepAwake` on an unheld tag is a native call whose + * failure would read to the page as a wake lock it could not drop. And a page session that ends + * with a tag still held has it given back for it — the page is a document that can navigate, fault + * or be swiped away mid-dictation, and nothing else would ever call `deactivate`, so the screen + * would stay awake for the app's lifetime. + * + * So the set means "the device still has this tag", not "the page asked for it": a deactivation the + * device refused leaves the tag recorded, because the page's owner queues exactly that failure for + * a retry and the retry has to reach the device. + */ +export type WakelockDevice = { + readonly activate: (tag: string) => Promise + readonly deactivate: (tag: string) => Promise +} + +export type NativeWakelockServer = { + readonly serve: (params: unknown) => Promise<{ active: boolean }> + /** Gives back every tag this session still holds. The page session's end and the screen's + * unmount both call it, exactly as they do for a staged media handle and a live microphone. */ + readonly dispose: () => void +} + +export function createNativeWakelockServer(device: WakelockDevice): NativeWakelockServer { + const held = new Set() + /** + * One chain per tag, because `held` is read and written across an await. + * + * A release that arrives while its own activate is still in flight would otherwise read the set + * before the activate had recorded anything, find nothing, deactivate nothing and report the tag + * off — and then the activate lands and the device holds a tag the page has already said it does + * not want. Per tag rather than one chain for the server: a device call that hangs on one + * dictation's tag must not hold up another's. + */ + const queues = new Map>() + let disposed = false + + function enqueue(tag: string, action: () => Promise): Promise { + const previous = queues.get(tag) ?? Promise.resolve() + // On both settle paths: an activate the device refused must not wedge every later release. + const run = previous.then(action, action) + const settled = run.then( + () => undefined, + () => undefined + ) + queues.set(tag, settled) + void settled.then(() => { + // Dropped once nothing is behind it, so a screen's worth of dictations does not accumulate. + if (queues.get(tag) === settled) { + queues.delete(tag) + } + }) + return run + } + + async function set(active: boolean, tag: string): Promise<{ active: boolean }> { + if (active) { + await device.activate(tag) + // Recorded the moment the device has it, before anything else here can fail. The set means + // "the device still has this tag", and the compensating release below is the one path that + // could leave a tag on with nothing recorded. + held.add(tag) + if (!disposed) { + return { active: true } + } + // The session can end between the call and its reply — the page is a document that can be + // swiped away mid-dictation — and a tag recorded after that dispose is held by nobody: + // `dispose` has already walked the set and nothing will walk it again. So it is given back + // here instead, and the page is told it is not held. A refusal rejects rather than reporting + // a tag the device still holds as free, which is what lets the caller's retry path run. + await device.deactivate(tag) + held.delete(tag) + return { active: false } + } + if (held.has(tag)) { + // Deleted only once the device has really dropped it. A refusal rejects out of here, which + // is how the page's owner learns to queue a retry — and that retry arrives as another + // `active: false`, so the tag has to still be recorded or it would answer "not held" + // without calling anything and leave the native tag on for the life of the app. + await device.deactivate(tag) + held.delete(tag) + } + return { active: false } + } + + return { + // Parsed before the queue, so a malformed request is refused rather than waiting behind a tag. + serve: async (params) => { + const { active, tag } = wakelockSetParamsSchema.parse(params) + return await enqueue(tag, () => set(active, tag)) + }, + dispose: () => { + disposed = true + for (const tag of Array.from(held)) { + // Quiet, for the reason every other dispose here is: this runs while a screen is going + // away, and a device that would not drop a tag is not something the page can be told about. + // Forgotten only on success, so one this device refused stays recorded and a later release + // still reaches it. Queued behind that tag's own operations rather than racing them, and + // re-reading the set once it runs: a release already in flight may have given it back, and + // deactivating an unheld tag is a native call this module does not make. + void enqueue(tag, async () => { + if (!held.has(tag)) { + return + } + await device.deactivate(tag) + held.delete(tag) + }).catch(() => undefined) + } + } + } +} diff --git a/mobile/src/platform/use-native-device-verbs.test.tsx b/mobile/src/platform/use-native-device-verbs.test.tsx index 52894e84fae..43dac446e7a 100644 --- a/mobile/src/platform/use-native-device-verbs.test.tsx +++ b/mobile/src/platform/use-native-device-verbs.test.tsx @@ -1,12 +1,14 @@ /** The one handler the host dispatches to: which verb reaches which device half, and for how long. */ import type { ReactElement } from 'react' import { act, create } from 'react-test-renderer' -import { describe, expect, it, vi } from 'vitest' +import { beforeEach, describe, expect, it, vi } from 'vitest' const device = vi.hoisted(() => ({ clipboard: { getStringAsync: vi.fn(() => Promise.resolve('on the pasteboard')) }, picker: { launchImageLibraryAsync: vi.fn(() => Promise.resolve({ canceled: true })) }, - deleted: new Array() + deleted: new Array(), + /** Every wake-tag call the device saw, as `+tag` and `-tag`, so an unreleased tag is visible. */ + wakeTags: new Array() })) vi.mock('expo-clipboard', () => ({ @@ -17,6 +19,24 @@ vi.mock('expo-clipboard', () => ({ vi.mock('expo-document-picker', () => ({ getDocumentAsync: () => Promise.resolve({ canceled: true }) })) +vi.mock('@orca/expo-two-way-audio', () => ({ + addExpoTwoWayAudioEventListener: () => ({ remove: () => {} }), + initialize: () => Promise.resolve(true), + requestMicrophonePermissionsAsync: () => + Promise.resolve({ granted: true, canAskAgain: true, status: 'granted', expires: 'never' }), + tearDown: () => {}, + toggleRecording: () => true +})) +vi.mock('expo-keep-awake', () => ({ + activateKeepAwakeAsync: (tag: string) => { + device.wakeTags.push(`+${tag}`) + return Promise.resolve() + }, + deactivateKeepAwake: (tag: string) => { + device.wakeTags.push(`-${tag}`) + return Promise.resolve() + } +})) vi.mock('expo-image-picker', () => ({ launchImageLibraryAsync: device.picker.launchImageLibraryAsync })) @@ -95,3 +115,37 @@ describe('the device handler the shell hands its host', () => { unmount() }) }) + +describe('the wake tag a page session takes', () => { + beforeEach(() => { + device.wakeTags.length = 0 + }) + + it('is given back when the session ends, not left holding the screen awake', async () => { + const first = mount('session-a') + await expect( + first.serve('native.wakelock.set', { active: true, tag: 'orca-a' }) + ).resolves.toEqual({ active: true }) + // The page is a document that can navigate, fault or be swiped away mid-dictation, so a tag it + // took and never released would keep the screen awake for the app's lifetime. + first.unmount() + await Promise.resolve() + expect(device.wakeTags).toEqual(['+orca-a', '-orca-a']) + }) + + it('leaves the next session nothing of the last one to release', async () => { + const first = mount('session-a') + await first.serve('native.wakelock.set', { active: true, tag: 'orca-a' }) + first.unmount() + await Promise.resolve() + device.wakeTags.length = 0 + const second = mount('session-b') + // A tag the previous session held is the previous session's; this one asking for it back must + // not reach the device, and must not report it as held either. + await expect( + second.serve('native.wakelock.set', { active: false, tag: 'orca-a' }) + ).resolves.toEqual({ active: false }) + expect(device.wakeTags).toEqual([]) + second.unmount() + }) +}) diff --git a/mobile/src/platform/use-native-device-verbs.ts b/mobile/src/platform/use-native-device-verbs.ts index 7d794d43eb8..c0774ca9529 100644 --- a/mobile/src/platform/use-native-device-verbs.ts +++ b/mobile/src/platform/use-native-device-verbs.ts @@ -1,17 +1,20 @@ -import { useMemo } from 'react' +import { useEffect, useMemo } from 'react' import type { BridgeNativeVerb } from '../mobile-web-shell/bridge/bridge-native-verbs' import { useMediaHandleRegistry } from '../mobile-web-shell/use-media-handle-registry' +import { createNativeAudioCapture } from './native-audio' +import { nativeAudioDeviceEngine, nativeWakelockDevice } from './native-audio-device' import { serveNativeClipboardVerb } from './native-clipboard' import { createNativeMediaVerbServer } from './native-media' import { discardStagedMedia, nativeMediaDeviceDeps } from './native-media-device' +import { createNativeWakelockServer } from './native-wakelock' /** * Every `native.` verb this device serves, behind the one function the host dispatches to. * - * Built here rather than in the screen because the media verbs are stateful where the clipboard - * ones are not: they hold staged files, and the registry that owns them has to be born and swept - * with the page session. The screen passes a session id and gets a handler whose lifetime already - * matches it. + * Built here rather than in the screen because most of them are stateful where the clipboard ones + * are not: the media verbs hold staged files, the audio verbs hold a live microphone and the wake + * lock holds a tag, and all three have to be born and released with the page session. The screen + * passes a session id and gets a handler whose lifetime already matches it. */ export function useNativeDeviceVerbs( sessionId: string | null @@ -21,11 +24,22 @@ export function useNativeDeviceVerbs( () => createNativeMediaVerbServer(nativeMediaDeviceDeps(registry)), [registry] ) + // Keyed on the session for the registry's reason: a new page session is a new document, and a + // microphone the previous one left open is nobody's to stop but this seam's. + const audio = useMemo(() => createNativeAudioCapture(nativeAudioDeviceEngine), [sessionId]) + const wakelock = useMemo(() => createNativeWakelockServer(nativeWakelockDevice), [sessionId]) + useEffect(() => () => audio.dispose(), [audio]) + useEffect(() => () => wakelock.dispose(), [wakelock]) return useMemo( - () => (verb, params) => - verb === 'native.clipboard.write' || verb === 'native.clipboard.read' - ? serveNativeClipboardVerb(verb, params) - : serveMedia(verb, params), - [serveMedia] + () => (verb, params) => { + if (verb === 'native.clipboard.write' || verb === 'native.clipboard.read') { + return serveNativeClipboardVerb(verb, params) + } + if (verb === 'native.wakelock.set') { + return wakelock.serve(params) + } + return verb.startsWith('native.audio.') ? audio.serve(verb, params) : serveMedia(verb, params) + }, + [audio, serveMedia, wakelock] ) } diff --git a/mobile/src/session/mobile-dictation-mic-control.web.test.tsx b/mobile/src/session/mobile-dictation-mic-control.web.test.tsx new file mode 100644 index 00000000000..6549cd594a8 --- /dev/null +++ b/mobile/src/session/mobile-dictation-mic-control.web.test.tsx @@ -0,0 +1,236 @@ +/** + * The mic control on the page, driven by the real seam over the real shell. + * + * Ruling 4's negative proof, now with something behind it: on a route the shell did not grant the + * audio verbs the control has to report the refusal and come back, rather than crash or sit on + * "Starting voice dictation" forever. On a route that was granted them it has to reach recording — + * which is the whole of the item, observed where a user would see it. + * + * Mounted against `dictation-capture.web.ts` rather than the native sibling, because that is the + * substitution the web build makes: the bundler resolves `.web.ts` first and vitest resolves the + * native file. Everything else is real — the port pair, the verb table, the shell's ring. + */ +import { createElement, type ReactElement, type ReactNode } from 'react' +import { act, create, type ReactTestInstance } from 'react-test-renderer' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +// The renderer needs host elements, not the real platform: what this observes is the label the +// control carries and the tap that reaches its handler, neither of which is native. +vi.mock('react-native', () => ({ + ActivityIndicator: (props: Record) => + createElement('rn-activity-indicator', props), + AppState: { currentState: 'active', addEventListener: () => ({ remove: () => {} }) }, + Platform: { OS: 'ios' }, + Pressable: (props: { children?: ReactNode }) => + createElement('rn-pressable', props, props.children) +})) +vi.mock('lucide-react-native', () => ({ + ImagePlus: (props: Record) => createElement('lucide-image-plus', props), + Mic: (props: Record) => createElement('lucide-mic', props) +})) + +// The page's capture seam, which is what the web build resolves. +vi.mock( + '../platform/dictation-capture', + async () => await import('../platform/dictation-capture.web') +) + +// The provider module re-exports the screen hooks, and reaching the real ones imports the Expo +// runtime this test does not have. Nothing below calls one. +vi.mock('../transport/host-client-hooks', () => ({ + useDisconnectHostClient: () => () => {}, + useForceReconnect: () => () => Promise.resolve(), + useForgetHostClient: () => () => {}, + useHostClient: () => ({ client: null, clientId: null, state: 'disconnected' }), + usePrimeHosts: () => () => {}, + useRefreshHostClient: () => () => {} +})) + +import { RpcClientProvider } from '../transport/client-context.web' +import { + createFakeBridgePortPair, + type BridgePortPair +} from '../mobile-web-shell/bridge/bridge-port-pair-test-harness' +import { createNativeAudioCapture, type NativeAudioEngine } from '../platform/native-audio' +import { createNativeWakelockServer } from '../platform/native-wakelock' +import { useMobileDictation } from '../hooks/use-mobile-dictation' +import { MobileTerminalInputActions } from './MobileTerminalInputActions' +import type { BridgeNativeVerb } from '../mobile-web-shell/bridge/bridge-native-verbs' + +/** Every message that reached the composer's own error handler, which is what it toasts. */ +const reported: string[] = [] + +/** The four verbs served by the real handlers over an engine that opens and produces no audio. */ +function createAudioShell(): (verb: BridgeNativeVerb, params: unknown) => Promise { + const engine: NativeAudioEngine = { + requestPermission: async () => 'granted', + open: async (sampleRate) => ({ opened: true, sampleRate }), + begin: () => true, + end: () => {}, + onMicrophoneData: () => ({ remove: () => {} }), + onInterruption: () => ({ remove: () => {} }) + } + const capture = createNativeAudioCapture(engine) + const { serve: wakelock } = createNativeWakelockServer({ + activate: async () => undefined, + deactivate: async () => undefined + }) + return (verb, params) => + verb === 'native.wakelock.set' ? wakelock(params) : capture.serve(verb, params) +} + +function Composer({ pair }: { pair: BridgePortPair }): ReactElement { + const dictation = useMobileDictation({ + client: pair.client, + enabled: true, + onTranscript: () => {}, + onError: (error) => reported.push(error.message) + }) + return ( + {}} + onAttachFile={() => {}} + onDictationToggle={() => { + // The composer's own handler, as `use-mobile-session-native-chat-dictation.ts` writes it: a + // refused start is a toast, never a throw into render. + void dictation.start().catch((error: unknown) => { + reported.push(error instanceof Error ? error.message : String(error)) + }) + }} + onDictationPressIn={() => {}} + onDictationPressOut={() => {}} + onDictationCancel={() => { + void dictation.cancel() + }} + /> + ) +} + +type MicControl = { + readonly label: () => string + readonly tap: () => Promise +} + +/** The mic button, found by the label it carries in every state rather than by position. */ +function micOf(root: ReactTestInstance): ReactTestInstance { + const found = root.findAll( + (node) => + typeof node.type === 'string' && + typeof node.props.accessibilityLabel === 'string' && + node.props.accessibilityLabel.includes('voice dictation') + ) + const mic = found[0] + if (found.length !== 1 || mic === undefined) { + throw new Error(`expected one mic control, found ${found.length}`) + } + return mic +} + +async function mount(pair: BridgePortPair): Promise { + await pair.flush() + const held: { tree: ReturnType | null } = { tree: null } + await act(async () => { + held.tree = create( + + + + ) + }) + const rendered = held.tree + if (rendered === null) { + throw new Error('nothing mounted') + } + return { + label: () => String(micOf(rendered.root).props.accessibilityLabel), + tap: async () => { + await act(async () => { + micOf(rendered.root).props.onPress() + }) + // The tap crosses the bridge, the shell answers, and the desktop answers what was forwarded. + for (let round = 0; round < 4; round += 1) { + await act(async () => { + await pair.flush() + for (const request of pair.rpc.requests.splice(0)) { + request.resolve({ id: 'desktop', ok: true, result: {} }) + } + await pair.flush() + }) + } + } + } +} + +beforeEach(() => { + reported.length = 0 +}) + +afterEach(() => { + vi.useRealTimers() +}) + +describe('the mic control on a page the shell did not grant audio', () => { + it('reports the shell refusal and comes back to its resting label', async () => { + const pair = createFakeBridgePortPair({ + serveNativeVerb: createAudioShell(), + routeGrants: ['navigate', 'storage'] + }) + const mic = await mount(pair) + expect(mic.label()).toBe('Start voice dictation') + await mic.tap() + // The refusal is what the composer toasts, naming the verb rather than a blank failure. + expect(reported).toEqual(['this shell did not grant native.audio.start']) + // And the control is usable again: a page stuck on "Starting" has no way back to idle. + expect(mic.label()).toBe('Start voice dictation') + }) + + it('sends no frame for a verb it was never granted', async () => { + const pair = createFakeBridgePortPair({ + serveNativeVerb: createAudioShell(), + routeGrants: ['navigate', 'storage'] + }) + const mic = await mount(pair) + await mic.tap() + expect( + pair + .readToShell() + .filter((frame) => frame.type === 'request' && frame.method.startsWith('native.audio.')) + ).toEqual([]) + }) +}) + +describe('the mic control on a page the shell did grant audio', () => { + it('reaches recording, with no refusal reported', async () => { + const pair = createFakeBridgePortPair({ serveNativeVerb: createAudioShell() }) + const mic = await mount(pair) + expect(mic.label()).toBe('Start voice dictation') + await mic.tap() + expect(reported).toEqual([]) + expect(mic.label()).toBe('Stop voice dictation') + }) + + it('opens the capture and takes the wake tag through the shell', async () => { + const pair = createFakeBridgePortPair({ serveNativeVerb: createAudioShell() }) + const mic = await mount(pair) + await mic.tap() + const verbs = pair + .readToShell() + .flatMap((frame) => + frame.type === 'request' && frame.method.startsWith('native.') ? [frame.method] : [] + ) + expect(verbs).toContain('native.audio.start') + expect(verbs).toContain('native.wakelock.set') + // And the desktop was told, which is what makes the recording a session rather than a mic. + expect( + pair + .readToShell() + .some((frame) => frame.type === 'request' && frame.method.startsWith('speech.dictation.')) + ).toBe(true) + }) +}) diff --git a/mobile/web-entry/web-overrides.json b/mobile/web-entry/web-overrides.json index 91a44ce84fe..6730ea10c17 100644 --- a/mobile/web-entry/web-overrides.json +++ b/mobile/web-entry/web-overrides.json @@ -7,7 +7,7 @@ }, { "file": "packages/expo-two-way-audio/src/ExpoTwoWayAudioModule.web.ts", - "reason": "Vendored with the module, not added for Route A. The dictation hook imports @orca/expo-two-way-audio, whose native module is a Swift/Kotlin JSI binding with no browser counterpart; the web file answers the same surface with denied microphone permission and no playback." + "reason": "Vendored with the module, not added for Route A. Nothing on the page reaches it any more: dictation's capture went behind src/platform/dictation-capture.web.ts, which asks the shell for the microphone, so the only importer of @orca/expo-two-way-audio is the native half of that seam. The file stays because it ships with the vendored module, and it answers denied microphone permission and no playback, which is what a browser outside the shell can honestly say." }, { "file": "src/navigation/route-handoff.web.ts", @@ -113,6 +113,10 @@ "file": "src/terminal/terminal-webview-html.web.ts", "reason": "The native file composes the whole WebView document, which splices in the 612 KiB minified xterm engine string. On the page that string is unusable — the shell's CSP is script-src 'self' with frame-src 'none', so there is no nested document to load it into — and it would be the largest single module in the session route's closure. The web file answers the caret options, the markup and the stylesheet, which is everything the page mounts, and nothing else; mobile-web-terminal-engine-closure.test.mjs is the fence." }, + { + "file": "src/platform/dictation-capture.web.ts", + "reason": "Dictation's capture seam. The native file holds the microphone through @orca/expo-two-way-audio and the wake tag through expo-keep-awake, neither of which a browser has; this one asks the shell for both over native.audio.start|read|stop and native.wakelock.set, draining the shell's ring on a timer and raising each reply as the events the engine emits directly. The flow above the seam is one file on both hosts." + }, { "file": "src/platform/media-picker.web.ts", "reason": "expo-image-picker and expo-document-picker are native modules whose import runs a codegen lookup that throws in a browser, and the route manifest imports every route, so one of them in a page closure takes the whole bundle down rather than one picker. This one asks the shell through native.media.pick, reads the bytes back a chunk at a time over native.media.read because a picked image reaches 18 MiB raw against an 8 MiB reply ceiling, and releases every handle it was handed, including the ones its caller never took. The pasteboard is not here: clipboard.web.ts owns it on both platforms and reaches the same verbs for an image."