mirror of
https://github.com/stablyai/orca.git
synced 2026-10-19 00:03:20 +00:00
ishihara takashiandJinjing
6ee2af357b
* Add in-place screenshot markup for the browser pane Adds a Draw button to the browser toolbar (local and remote/SSH panes) that freezes the visible viewport into a still image and overlays a drawing canvas. Users can mark it up with pen, highlighter, arrow, rectangle, ellipse, and text (color, thickness, undo/redo), then copy the composited PNG to the clipboard to paste into their agent — reusing the existing clipboard screenshot path, so it works for local and remote agents alike. The drawing layer is a renderer-side canvas, so the base image is the only environment-specific piece: local panes use webview.capturePage(), remote panes snapshot the already-displayed screencast frame. Drawing model, compositing, and shape rendering are isolated, unit-tested modules. * Fix browser markup clipboard, transparency, text input, and toggle - Copy now always produces a PNG (the clipboard handler accepts PNG only; a JPEG fallback silently produced an empty clipboard) and targets the clipboard size limit so realistic viewports stay full-resolution. - Flatten the captured base image onto opaque white so transparent page backgrounds no longer ghost the live view through the frozen backdrop. - Keep markup text-input keystrokes local (focus the field explicitly and stop propagation) so the browser pane's global handlers can't swallow them. - Make the Draw toolbar button toggle markup mode off on a second click. * Add markup text sizing, object re-editing, and text-input fixes - Add a font-size control (text tool) to the markup toolbar. - Add a Select tool to re-edit committed shapes: click to select, drag to move, change color/width/font-size of the selection, Delete to remove, and double-click text to re-edit its content. Undo/redo now snapshots the whole shape list so every edit (not just adding) is reversible. - Fix the text input: transparent ink-colored field (no dark theme box over the screenshot) and ignore Enter during IME composition so Japanese conversion no longer commits the annotation early. - Split the grown overlay logic into focused modules (editor hook, pointer + keyboard hooks, canvas renderer, document edits, hit-testing) to stay within the file-size budget. * Improve markup text commit and move toolbar to the bottom - Clicking elsewhere while a text box is open now commits the text and switches to the Select tool, instead of discarding it (the input unmounted before its blur fired) and opening a new box at the click. - Move the drawing toolbar to the bottom (stacked above the actions bar) and open the color/size popover upward. * Fix markup text re-editing alignment, doubling, and selection bounds - Hide the text shape being re-edited from the canvas so the live input isn't doubled by its committed render. - Drop the input's padding/border and use the same font as the canvas so the typing preview sits exactly where the text renders (no shift); field-sizing keeps the box hugging the text. - Estimate text width with full-width (CJK) awareness so the selection box and hit bounds track Japanese text instead of clipping it. - Move the font-size control out of the color popover into its own toolbar button showing the current size. * Unify markup text selection and edit framing Draw the selection box on the canvas even while a text shape is being re-edited (the shape stays hidden so it isn't doubled), and strip the box off the text input so it renders only the editable glyphs. The frame is now identical whether selecting or editing, and the text overlays the same position with the same font, so there's no shift between the two. * Align markup text-edit baseline with the committed render Force the text input's line-height to 1 and zero its height/padding so its top edge matches the canvas textBaseline:'top' render — the editing text no longer sits slightly lower than the committed/selected text. * Address review: markup hit-testing, clear reset, and async guards - Hit-test highlights against their rendered (4x) thickness so the select tool doesn't miss large highlight strokes. - Reset transient editor state (pending text, selection, in-progress drag) on Clear all so a pending input blur can't re-add text and no stale selection lingers over the emptied canvas. - Guard markup completion with the capture token so a stale onDeliver can't reset or error a session the user already cancelled or restarted. - Skip the grab-element keyboard shortcut while the markup overlay is open, matching the already-disabled grab toolbar buttons. - Localize the markup error messages and surface them via a toast (the controller's error was otherwise never shown). * Trim browser markup to a draw-only tool and fix two delivery bugs Reshape the screenshot-markup overlay to match its actual use — a throwaway scribble the user copies once into their agent — and drop the re-editable vector-editor machinery that made it an outlier in scope. - Remove the select/move/restyle/re-edit subsystem: hit-testing, per-shape document edits, drag preview, selection frame, double-click text re-edit, Delete-to-remove, and the CJK text-width heuristic / multi-line text plumbing that only fed selection bounds. Keep pen, highlighter, arrow, rect, ellipse, and text with color/thickness/font-size and undo/redo/clear. - Fix silent data loss on HiDPI: the compositor budgeted only bytes, so a large Retina viewport could produce a PNG under the byte limit but over the clipboard handler's pixel ceiling, which the handler drops silently while the UI still reported success. Composite now scales down to fit maxPixels too. - Fix a stranded UI after a capture failure: the controller stayed "active" with no overlay and an inert Escape. It now returns to idle (capture failure) or back to the drawing surface (compose failure) after the error toast. Net ~660 fewer lines and 3 fewer modules; delivery (freeze → composite PNG → existing clipboard-image paste) is unchanged. Unit tests updated. * Add a one-time discovery hint for the browser Draw button Highlight the new screenshot-markup Draw button the first time it's usable so users — including existing ones — notice the tool. A localStorage-gated, show-once tooltip + highlight ring, independent of the browser contextual tour (which is capped at three steps and only shows to users who haven't seen it). - New use-markup-draw-hint hook: opens once per install when the button is enabled and idle, auto-dismisses, and dismisses on click; skips cleanly when storage is unavailable. - MarkupDrawButton forces its tooltip open with the discovery copy and rings the button while the hint shows. - Localize the hint (drawHint) across en/es/ja/ko/zh; normalize the markup key order alphabetically. * Align the markup copy button label with "Copy Screenshot" The browser pane's two image-copy actions read inconsistently in English — "Copy Screenshot" (the grab flow) vs "Copy markup" (the overlay). Match the new markup button to the existing one: "Copy Markup". English only; other locales already use their own sentence-case convention consistently, so they're unchanged. * Drop the markup Draw hint auto-timeout; dismiss on act only The one-time Draw-button hint no longer auto-dismisses on a timer the user might miss. It now stays open until the user clicks Draw or the button stops being usable (grab started, markup open, or a blank tab), which also fixes a case where losing eligibility mid-hint left the forced-open tooltip stuck over a disabled button. * Optimize markup rendering and harden compose delivery Performance: - Rasterize committed shapes once into an offscreen layer; per-frame paint blits that layer and draws only the in-progress shape, so a fast pointermove stream never re-strokes the whole scene. - Encode the composite PNG off the main thread via canvas.toBlob (a large synchronous toDataURL froze the renderer), with a rAF yield so the "composing" UI paints first. Correctness / review fixes: - Don't overwrite the clipboard after cancel: re-check the capture token between the (now async) compose and onDeliver, closing the window where Escape during composing still delivered the markup. - Keep the offscreen layer and visible canvas in lockstep by threading one measured devicePixelRatio through both paints, and re-measure on window resize so a monitor move (dpr change without a CSS-box change) repaints at the new scale. Blit stretched to the canvas box so a transient mismatch never clips. - Drop the stale "or JPEG fallback" note; compose is PNG-only. * Fix test fixtures for markup pixel-budget clamping to use a truly oversi The old 3840×2160×dpr2 fixture (33.2M px) was already under the 32×1024×1024 (~33.55M) ceiling, so the "never over budget" assertions weren't exercising the clamp path. Switch to 4000×3000×dpr2 (48M px), which genuinely exceeds the ceiling. --------- Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>