mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Neil
2167cd2994
* test(monaco): drive the real Monarch tokenizer instead of walking rule tables
* fix(monaco): stop a truncated JSONL record poisoning every record after it
`@string` was pushed unconditionally, and Monarch state survives the line
break, so one truncated record left every later record inside the string state
— the whole rest of the file rendered as a single string. A truncated record is
a normal way for a .jsonl log to end.
Gate the push on a lookahead proving the closing quote is on this line, and
consume an unterminated remainder as `string.invalid` without pushing anything.
Measured against the real tokenizer over realistic JSONL. Two alternatives were
rejected: collapsing the string into one regex fixes the poisoning but loses
every `string.escape` token on well-formed records, and `includeLF` does not
fix the primary case at all (the `[^"\\]+` content rule swallows the newline
before an EOL rule can match). This candidate's token stream is byte-identical
to the previous grammar on every well-formed record — the existing inline
snapshot did not move — and is at or faster than it on 20k pathological lines.
* test(monaco): pin monaco's own mdx grammar as the embed-recursion regression
Upstream's shipped mdx grammar enters a `js` embed on every `{` and pops
on `}` with no budget, so it reproduces the unbounded embed-entry
recursion exactly — evidence the shape is monaco's, not something Orca's
grammars invented, and a tripwire for a monaco upgrade that changes it.
The same file proves an Orca grammar stays inside the budget under the
identical line. Measured here: 500 interpolations -> 500 frames, no error;
3000 chars -> RangeError at 945 frames. The ceiling is runtime-dependent,
so the test asserts the failure, not the number.
Adds a regression for astro's `^`-anchored frontmatter pop rule
(monaco-editor#1127) so an indented or trailing `---` cannot close the
fence early, de-duplicates the line-cap constant onto the budget module's
`MAX_TOKENIZATION_LINE_LENGTH`, and renames the recursion suite: embeds
cannot nest, so "embedded recursion depth" described the wrong thing.