6 Commits

Include renames

Author	SHA1	Message	Date
Neil	1924c8f5b1	feat(perf): lint repeated sort setup and schedule regression contracts (#18822) ... * feat(perf): audit comparator setup and schedule performance contracts * test(sqlite): close readers after expected busy failures * ci(perf): trigger contract workflow on the contract files themselves Without these paths a contract rename lands green on PR CI and only breaks the next nightly, where nobody owns the failure. Also run the OS-independent source audit once instead of on all three runners.	2026-09-05 13:56:06 -07:00
Brennan BensonandMerge Sim	f4c2821167	refactor(agent-session-journal): move the session journal onto SQLite (#18652) ... * refactor(agent-session-journal): move the session journal onto SQLite The agent-session journal kept its state in three hand-rolled file formats: an append-only `log.jsonl` with torn-tail repair, a `snapshot.json` holding folded state plus a retained tail, and byte-quarantine files for anything unreadable. This replaces all of it with one SQLite database per session — `journal.db` beside the existing `blobs/` store — using the in-house adapter and the open/pragma/migrate/harden pattern the orchestration database already follows. Two tables: `journal_rows` (the append-only log, keyed by `(session_id, epoch, seq)`) and `journal_sessions` (the derived projection, upserted in the SAME transaction as every insert). Rows stay JSON in one column, so the row schema, the version upcast chain, and the reducer survive byte for byte — `journal-reducer.test.ts` and four other suites pass unchanged and are the regression proof. Deleted: `journal-log-file.ts`, `journal-compaction.ts`, `journal-corruption-quarantine.ts`, and the public `compact()` / `compactionBoundary` / `autoCompact` members, none of which had a non-test caller. Existing `log.jsonl` / `snapshot.json` journals are deliberately abandoned. No importer: a session created on the old path stops working, which is acceptable because the feature is off by default. ## The physical quota is repriced, because SQLite does not charge like a file The 256 MiB per-session bound is unchanged, but the arithmetic under it could not survive: SQLite grows the database in pages and the WAL in frames, and the checkpoint that copies the WAL forward holds the same pages in both files at once, so a transaction's peak is about twice its content. Admission now charges the candidate transaction's own measured page cost, validated against a sweep that runs as a regression test (`journal-database-space.test.ts`) rather than derived from reasoning about the allocator. Four things are load-bearing rather than tuning, each measured: - `auto_vacuum = INCREMENTAL` must be set BEFORE `journal_mode = WAL`. Set it after and it is ignored with no error, reclamation silently becomes a no-op, and the file never shrinks again. Both halves are asserted. - `wal_autocheckpoint = 0` plus an explicit `wal_checkpoint(TRUNCATE)` at the end of every write path, so the one moment the same pages live in two files is a moment the charge accounts for. - Reclamation runs in bounded chunks. A single unbounded `incremental_vacuum` took a 252 MB directory to 504 MB — the reclamation added to defend the bound would have breached it. `PRAGMA incremental_vacuum(N)` also frees exactly one page unless it is stepped to completion, which no size assertion catches, so the freed page count is asserted directly. - A blocked checkpoint leaves the WAL on disk together with the database growth it already copied, so admission charges that deferred copy explicitly. The term is zero whenever the last checkpoint succeeded, so the uncontended path admits and refuses an identical set. The epoch discard is `DELETE FROM journal_rows` with no WHERE clause, which takes SQLite's truncate optimization: measured at ~0.26% of the database in WAL bytes where the `WHERE session_id = ?` form rewrote every emptied leaf at up to 99%. One database per session is what makes the unqualified form correct. An open, empty journal costs 57,344 bytes before a single row exists, so a configured quota below `JOURNAL_MIN_SESSION_BYTES` now fails loudly at open with the existing `journal_bound_exceeded` instead of as a run of identical append failures. No production caller configures one; the affected surface is test fixtures, rescaled to the smallest value that restores what each case proves. ## One deliberate behaviour change Compaction was the only mechanism that shed bytes inside an epoch, and the write path called it precisely so an append at the bound was not refused. The SQLite-shaped replacement — a bounded prefix delete — cannot be used: with the snapshot gone the surviving rows ARE the state, so dropping the oldest of them loses the oldest transcript silently at the next reopen. So no row is ever shed inside an epoch, and a session whose row bytes alone reach the bound now refuses every append where it previously compacted and continued. A loud typed refusal beats silent data loss. What still sheds is unreferenced BLOB bytes — the dominant and unbounded byte source — on the same write-path hook. The escape from the hard stop is the fold that already exists, `replaceEpochItems`, which now actually returns bytes to the filesystem instead of leaving them on the freelist. The prune's protected set is a union of live reducer digests AND the candidate row's own digests, including those cited only by a nested lifecycle-batch mutation. Content addressing never rewrites a digest already on disk, so protecting live state alone deletes the blob the append is about to cite — a dangling reference that surfaces one reopen later as an empty expansion on an item the user can see. `journal-store-blob-budget.test.ts` pins it, and it goes red when the set is narrowed back. ## Handle ownership A file handle used to be opened and closed per append; a SQLite handle is held for the session's lifetime. Every path that can open a connection now has one owner: the open function owns its raw connection until it returns, the store owns its retained one and releases it in a new `close()`, and every other connection is closed by the call that opened it. The attach, recovery, eviction, map-overwrite and host-teardown paths close what they drop, and host teardown is failure-complete — the sink-barrier flush throws by design, so a trailing close statement would be skipped on exactly the path that leaks. `close()` has a stated contract: admission at enqueue and permanent, the close step on the same queue past that gate, one shared in-flight attempt, fulfilment terminal, and the release last and deliberately unguarded so a retry re-enters it. Guarding the release would skip it on retry, guaranteeing a permanent leak in exactly the case where it did not release. `journal_closed` joins the error union for a write after `close()`; no file outside the directory references any of these codes. * fix(agent-session-journal): make a COMMIT final, stop repairs deleting valid rows, and keep rejected closes retryable Six review findings on the SQLite journal migration. 1. A successful COMMIT is now the point of no return. The ordinary append, the epoch roll and the epoch replacement each adopt the committed row or epoch BEFORE any post-commit filesystem work; checkpoint, reclaim, blob prune and directory measurement run through `runJournalPostCommit`, which is best-effort by design and falls back to the transaction's own charge as a conservative footprint. Previously a post-COMMIT scan failure rejected a durable append and the next one reused its sequence, and a failed epoch housekeeping step left the store writing into a prefix already deleted. 2. Corruption repair preserves instead of destroying. A rejected suffix is copied into a new `journal_quarantine` table and removed from the live epoch in ONE transaction per chunk, charged against the session bound before a byte is written; a journal that cannot afford the copy refuses to open rather than falling back to deletion. The repair state is exposed as `journal.repair` and the rows are readable through `recoverQuarantinedRows()`, so Orca-owned submission, receipt and lifecycle identity survives a gap or a malformed row. 3. The physical charge covers the B-tree key payload. `session_id` and `epoch` are stored in both tables and both primary-key indexes and appear nowhere in `row_json`, so the journal boundary now bounds them and `journalTxnPhysicalCost` charges those bounds plus the projection upsert. The charge sweep runs the exact production transaction at maximum admitted key sizes. 4. A rejected `close()` no longer orphans its handle. Callers hand the journal to `agentSessionJournalCloseRetries` instead of swallowing the rejection, the attach map replacement is ABORTED when the previous journal will not close, host teardown retries what the registry holds, and a failed runtime teardown is retained so the next stop is a real retry. 5. `journalWalBytes()` returns zero only for ENOENT and propagates every other stat error, so admission and reclamation fail closed. 6. The WAL contention test closes the writer before removing its temp root and asserts the directory is removable once handles close. Regression coverage: post-commit divergence (4), corruption repair (5), key bounds (5), WAL stat (8), close retry (5), plus a runtime stop-retry case. Each fix was ablated on this head and the matching tests go red. * fix(agent-session-journal): anchor replay at sequence 1, make quarantine append-only, and charge it in bytes Three ways the corruption quarantine still lost rows it was written to keep. Replay validated contiguity from the first row that HAPPENED to remain, so an epoch missing only its sequence-1 row declared the leftovers contiguous and set no `truncateFrom`. The load was still corrupt, so recovery imported provider history and `replaceEpochItems` deleted every live row — including Orca-minted submission, receipt and lifecycle identity that no transcript can reconstruct, and that nothing had quarantined. Replay now anchors at sequence 1, so a missing epoch row rejects the whole surviving range before any replacement runs. `journal_quarantine` was keyed on `(session_id, epoch, seq)` and copied with `INSERT OR REPLACE`. A repair frees the sequences it removed and the live epoch reuses them, so a second repair in the same epoch silently deleted what the first preserved. The table is now keyed on a surrogate `quarantine_id`, the copy is a plain append, and `(epoch, seq)` is metadata; existing v1 databases are rekeyed in the migration that already bumps `user_version`. The admission charge read `length(row_json)`, which counts CHARACTERS for a TEXT value where `journalTxnPhysicalCost` expects physical UTF-8 bytes. A multibyte suffix was charged at up to a third of what it writes, which defeats the pre-write physical bound — over a megabyte on a maximum-size lifecycle batch. * fix(agent-session-journal): keep a repaired epoch anchored and stop the v1 quarantine migration doubling the file Replay validated numeric contiguity from sequence 1 but never that sequence 1 IS the epoch row. When the anchor was missing the repair set aside every surviving row, and if provider-history import then failed — a transcript that is temporarily gone is enough — the journal reopened as a clean, row-less epoch: an ordinary append took sequence 1, replay accepted it, read-restore published it as history, and automatic recovery never ran again while the user's real messages sat in quarantine. Replay now rejects an unanchored prefix, the open publishes an `unreconcilable_prefix` anchor for an epoch its repair emptied, and that anchor keeps reporting corrupt — so provider history is retried on every attach — until the timeline is rebuilt or the session writes content of its own. A repair also discloses rows it set aside when no line was unreadable at all, which is the case that removes the most. The v1 quarantine rekey copied every legacy row into the new table inside one transaction and dropped the old one. A quarantine holds whole rejected rows: a single 8 MiB row nearly doubled the database past the physical bound the open had already checked, the dropped pages only reached the freelist, and the next open refused the session it had just migrated. The v1 table is renamed and frozen instead, and reads take both generations. Table creation also moves inside the migration transaction, so a crash can no longer leave a v2-shaped database still reporting version 0 for an older build to write into. * fix(agent-session-journal): stop an empty provider transcript retiring the repair marker A transcript that exists but decodes to zero messages was imported as a success: the import published an empty `legacy_import` replacement that deleted the `unreconcilable_prefix` anchor and its disclosure, so the next probe read the session as clean and every later attach skipped provider recovery while the user's rows sat in quarantine for good. The import now leaves the epoch untouched when nothing decodes, reporting `replaced: false`, and recovery treats that like a transcript it could not read — the marker stands and a later attach with real history rebuilds the timeline. * style(agent-session-journal): merge the duplicate journal-database-space import * refactor(agent-session-journal): drop quarantine, byte bound, blob spill and rate limit Match what comparable implementations do: the journal is an unbounded append-only SQLite log with no side tables and no admission control. Corruption: the rejected suffix is DELETED rather than copied into a quarantine table. The load still reports `corrupt` and recovery still rebuilds the epoch from provider history, so the observable outcome is unchanged — only the preservation half is gone. The schema is back to one version with two tables; no v1 database exists outside unmerged commits of this branch, so the rekey migration and the two-generation read path go with it. Sequence-1 epoch anchoring and the empty-provider-transcript retry are kept: both are about the corrupt signal being correct. Size: no `maxSessionBytes`, so no page-cost arithmetic, reclaim band, incremental vacuum, lifecycle byte reservations or `journal_bound_exceeded`. `auto_vacuum` and `wal_autocheckpoint = 0` existed only to make a transaction's physical cost predictable for that charge; with the charge gone SQLite's default checkpointing is what the journal wants, and the explicit pre-close checkpoint is redundant with the one `db.close()` performs. WAL, `synchronous = FULL` and `busy_timeout` stay. Payloads: an oversized body is truncated at the existing inline cap with the existing marker and the remainder is discarded, bounded at the translation layer that already calls these helpers. The truncation point and message do not change; the content-addressed blob directory and all digest tracking do. Rate: no `maxAppendsPerWindow` and no `journal_rate_exceeded`. `JournalPayloadLimits` is now just the inline cap. * fix(agent-session-journal): mark a partial repair pending and bound multi-block tool input A repair that keeps its prefix had nothing durable to show for the suffix it deleted: a sequence gap costs no malformed row, so no disclosure is appended, and the surviving rows keep their epoch anchor. The next probe read a contiguous anchored prefix, called it clean, and the deleted stretch of timeline was never asked for again — silent loss, with the deletion already committed. The deletion now writes a `journal_repairs` marker in the SAME transaction, and replay keeps reporting corrupt while it stands. It retires under exactly the rule the emptied-epoch anchor takes: a fresh epoch carries the rebuild, or the session writes content of its own past the sequence the repair left free. The repair's own disclosure is not that content. Legacy import bounded a tool call's input only when it was the message's sole block; the multi-block path returned `tool-call` unchanged, so a mixed message from Claude, Grok or an omp execution cell persisted the whole input despite `inlineHeadBytes`. `boundBlock` now routes it through `boundToolInput`. Also drops canonical comments describing quarantine, snapshot files, blob storage and blob compaction — none of which exist any more. * fix(agent-session-wire): stop awaiting the synchronous journal probe loadJournal runs on a sync-database connection and returns JournalLoad | null, so both wire call sites were awaiting a non-Promise. The type-aware code-quality gate flags it; the native gate does not. --------- Co-authored-by: Merge Sim <sim@local>	2026-09-04 15:09:23 -07:00
Neil	9062494f9b	fix(ai-vault): stop a whole opencode.db failure reading as one skipped transcript (#16587) ... * fix(ai-vault): stop a whole opencode.db failure reading as one skipped transcript #15036 reported "1 transcript skipped / database is locked" with both Agent Session History scopes empty. Two separate defects. The panel counts every unkinded scan issue as a skipped transcript, so a failure that lost an entire *source* was reported as one lost *file*. The whole-database failure is now kinded `scope`, and an unknown `kind` from a newer host degrades to `scope` instead of failing validation and coming back unkinded — a mixed-version remote host previously turned a source-level failure into a phantom skipped transcript. The read also inherited sqlite3's 0 ms busy timeout, so a genuinely contended open failed in ~1 ms. It now opens once with a bounded timeout. No retry loop: sqlite's own busy handler already blocks and retries internally for the whole timeout, and WAL readers do not block on a writer at all (measured: 547/547 cross-process reads at timeout=0 while a writer held open transactions). Measured against a real Ubuntu-24.04 distro, Windows cannot take SQLite's file locks over \\wsl.localhost at all: an idle, never-WAL, nothing-attached database still answers SQLITE_BUSY, a 5 s busy timeout does not change it, and the identical bytes open fine once copied to local disk. So a lock-family error on that share never means "a writer holds it" and no timeout can help. The copy says so rather than sending the user after a write-ahead log that is not the problem. Restoring those sessions needs an in-distro read; that is a follow-up, and this PR no longer pretends a timeout will do it. immutable=1 is deliberately not used as a workaround: over the same share it opens and returns 100 of 150 rows, silently dropping everything still in the uncheckpointed -wal — in a history panel, exactly the newest sessions. * skip the provably futile busy wait on \\wsl.localhost paths	2026-08-26 23:37:16 -07:00
NeilandOrca	0fe04e2c91	perf(sqlite): cache prepared statements in SyncDatabase (#13769) ... prepare() recompiled every statement, so orchestration reads re-parsed the same SQL on the main thread. Adds a bounded LRU keyed by SQL, cleared on close() and before schema-changing exec(). Wildcard selects are excluded: node:sqlite builds the first post-schema-change row from stale column names, so a reused SELECT * can silently drop a freshly added column. PRAGMAs stay uncached. Co-authored-by: Orca <help@stably.ai>	2026-08-11 01:48:14 -07:00
Jinwoo HongandJinwoo-H	cf16eac7f6	fix(agent-hooks): keep Node 18 relay companion loadable (#13135) ... Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com>	2026-08-07 22:54:42 -07:00
Neil	1009ac9083	chore: update Electron to 42 (#3919)	2026-05-30 13:26:48 -07:00