mirror of
https://github.com/stablyai/orca.git
synced 2026-10-31 00:57:45 +00:00
fix(github): count gh pr and issue commands against the GraphQL rate limit (#28778)
* fix(github): count gh pr and issue commands against the GraphQL rate limit gh serves pr/issue/repo view-style subcommands over GraphQL, but the breaker filed every non-api command under core and only matched REST's "API rate limit exceeded" wording, so GraphQL exhaustion never tripped it. Map those subcommands to the graphql bucket, match GraphQL's "API rate limit already exceeded", and guard project-view issue close/reopen against the GraphQL quota. Fixes #28767 * test(github): drop redundant Date.now() args in graphql breaker test Keeps the flaky-test-pattern ratchet at its baseline. --------- Co-authored-by: Neil <neil@stably.ai>
This commit is contained in:
Žan Mlakar
Neil
@@ -9,14 +9,14 @@ This page covers the errors you’ll see most often and how to fix them.
|
||||
|
||||
## Quick triage
|
||||
|
||||
| What you see | Likely cause | First thing to try |
|
||||
| ----------------------------------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| “GitHub is rate-limiting requests” / “rate limit exceeded (core)” | GitHub REST (core) quota exhausted for your user | Wait for reset; stop extra `gh` / agent / Orca usage; check [Settings → Git → GitHub API Budget](/docs/settings) |
|
||||
| “GitHub authentication is unavailable” / `gh auth` prompts | `gh` not logged in, expired token, or bad `GITHUB_TOKEN` | `gh auth status`, then `gh auth login` |
|
||||
| “GitHub did not allow access” / HTTP 403 (not rate limit) | Missing scopes or no access to the repo | Re-auth with `repo` (and needed org SSO); confirm you can open the PR in the browser |
|
||||
| “repository is unavailable” / HTTP 404 | Wrong remote, private repo without access, or renamed repo | Check `git remote -v` and browser access |
|
||||
| “GitHub is unreachable” / timeouts | Network, proxy, VPN, or GitHub outage | Check [githubstatus.com](https://www.githubstatus.com/); retry off VPN |
|
||||
| “GitHub CLI is unavailable” | `gh` missing from PATH Orca uses | Install `gh` and restart Orca |
|
||||
| What you see | Likely cause | First thing to try |
|
||||
| -------------------------------------------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| “GitHub is rate-limiting requests” / “rate limit exceeded (core)” or “(graphql)” | GitHub REST (core) or GraphQL quota exhausted for your user | Wait for reset; stop extra `gh` / agent / Orca usage; check [Settings → Git → GitHub API Budget](/docs/settings) |
|
||||
| “GitHub authentication is unavailable” / `gh auth` prompts | `gh` not logged in, expired token, or bad `GITHUB_TOKEN` | `gh auth status`, then `gh auth login` |
|
||||
| “GitHub did not allow access” / HTTP 403 (not rate limit) | Missing scopes or no access to the repo | Re-auth with `repo` (and needed org SSO); confirm you can open the PR in the browser |
|
||||
| “repository is unavailable” / HTTP 404 | Wrong remote, private repo without access, or renamed repo | Check `git remote -v` and browser access |
|
||||
| “GitHub is unreachable” / timeouts | Network, proxy, VPN, or GitHub outage | Check [githubstatus.com](https://www.githubstatus.com/); retry off VPN |
|
||||
| “GitHub CLI is unavailable” | `gh` missing from PATH Orca uses | Install `gh` and restart Orca |
|
||||
|
||||
## Rate limits (most common)
|
||||
|
||||
@@ -24,13 +24,13 @@ GitHub gives each **authenticated user** a shared hourly budget. **Every tool on
|
||||
|
||||
### Buckets Orca cares about
|
||||
|
||||
| Bucket | What it covers | Typical limit (authenticated) |
|
||||
| --------------- | ---------------------------------------------------------------------------- | ----------------------------- |
|
||||
| **REST (core)** | Most PR/issue/API calls (`gh pr view`, checks metadata, many REST endpoints) | 5,000 / hour |
|
||||
| **GraphQL** | Project/Tasks and some richer PR queries | 5,000 points / hour |
|
||||
| **Search** | Search-driven lists | 30 / minute |
|
||||
| Bucket | What it covers | Typical limit (authenticated) |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
|
||||
| **REST (core)** | `gh api` REST endpoints: PR lookup for a branch, check runs, PR files, issue comments, labels | 5,000 / hour |
|
||||
| **GraphQL** | `gh pr view` / `list` / `checks` / `merge` / `edit`, `gh issue view` / `close` / `reopen` / `edit`, Project/Tasks queries | 5,000 points / hour |
|
||||
| **Search** | Search-driven lists | 30 / minute |
|
||||
|
||||
When a primary bucket is exhausted, GitHub returns HTTP **403** with a message like `API rate limit exceeded`. Orca classifies that as rate-limited, keeps the last known PR status when it can, and **stops spawning more `gh` calls** for a short window so a single limit doesn’t turn into a storm of failures.
|
||||
When a primary bucket is exhausted, REST calls fail with HTTP **403** and `API rate limit exceeded`, and GraphQL calls fail with `API rate limit already exceeded`. Orca classifies either as rate-limited, keeps the last known PR status when it can, and **stops spawning more `gh` calls** for a short window so a single limit doesn’t turn into a storm of failures.
|
||||
|
||||
### Why Settings can look “OK” while the PR panel is blocked
|
||||
|
||||
@@ -50,11 +50,14 @@ Also don’t confuse **GitHub API Budget** with **Claude / Codex / Grok usage**
|
||||
# Probe (does not consume quota; can look healthier than reality)
|
||||
gh api rate_limit --jq '.resources | {core, graphql, search}'
|
||||
|
||||
# Real REST call — this is what PR refresh depends on
|
||||
# Real REST call — PR refresh, check runs, PR files, and comments depend on this
|
||||
gh api user -i 2>&1 | head -40
|
||||
|
||||
# Real GraphQL call — PR details, PR lists, merges, and issue close/reopen depend on this
|
||||
gh api graphql -f query='{ viewer { login } }'
|
||||
```
|
||||
|
||||
If `gh api user` returns **403** with `API rate limit exceeded` and `X-Ratelimit-Remaining: 0`, the account is blocked for REST until `X-Ratelimit-Reset` (Unix epoch seconds).
|
||||
If `gh api user` returns **403** with `API rate limit exceeded` and `X-Ratelimit-Remaining: 0`, the account is blocked for REST until `X-Ratelimit-Reset` (Unix epoch seconds). If the GraphQL call fails with `API rate limit already exceeded`, the GraphQL bucket is the one exhausted.
|
||||
|
||||
### What usually burns the quota
|
||||
|
||||
|
||||
@@ -33,10 +33,14 @@ describe('classifyGhRateLimitBucket', () => {
|
||||
expect(classifyGhRateLimitBucket(['search', 'prs', '--author', 'x'])).toBe('search')
|
||||
})
|
||||
|
||||
it('classifies graphql and defaults everything else to core', () => {
|
||||
it('classifies graphql endpoints and GraphQL-backed subcommands, defaulting the rest to core', () => {
|
||||
expect(classifyGhRateLimitBucket(['api', 'graphql', '-f', 'query=…'])).toBe('graphql')
|
||||
expect(classifyGhRateLimitBucket(['pr', 'list', '--limit', '36'])).toBe('graphql')
|
||||
expect(classifyGhRateLimitBucket(['issue', 'close', '5', '--repo', 'a/b'])).toBe('graphql')
|
||||
expect(classifyGhRateLimitBucket(['repo', 'view', 'a/b', '--json', 'isFork'])).toBe('graphql')
|
||||
expect(classifyGhRateLimitBucket(['api', 'repos/a/b/pulls?per_page=36'])).toBe('core')
|
||||
expect(classifyGhRateLimitBucket(['pr', 'list', '--limit', '36'])).toBe('core')
|
||||
expect(classifyGhRateLimitBucket(['run', 'list', '--json', 'status'])).toBe('core')
|
||||
expect(classifyGhRateLimitBucket(['pr'])).toBe('core')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -69,6 +73,9 @@ describe('isGhPrimaryRateLimitStderr', () => {
|
||||
'gh: API rate limit exceeded for user ID 1775218. If you reach out to GitHub Support… (HTTP 403)'
|
||||
)
|
||||
).toBe(true)
|
||||
expect(
|
||||
isGhPrimaryRateLimitStderr('GraphQL: API rate limit already exceeded for user ID 1775218.')
|
||||
).toBe(true)
|
||||
expect(isGhPrimaryRateLimitStderr('You have exceeded a secondary rate limit.')).toBe(false)
|
||||
expect(isGhPrimaryRateLimitStderr('gh: Not Found (HTTP 404)')).toBe(false)
|
||||
})
|
||||
|
||||
@@ -78,13 +78,23 @@ function findGhApiEndpoint(args: readonly string[]): string | null {
|
||||
return null
|
||||
}
|
||||
|
||||
// gh serves these over GraphQL; a mixed REST trip (e.g. GHES `pr edit` reviewers) is corrected by the reset probe.
|
||||
const GH_GRAPHQL_SUBCOMMANDS = new Map<string, ReadonlySet<string>>([
|
||||
['issue', new Set(['close', 'edit', 'list', 'reopen', 'view'])],
|
||||
[
|
||||
'pr',
|
||||
new Set(['checks', 'close', 'create', 'edit', 'list', 'merge', 'ready', 'reopen', 'view'])
|
||||
],
|
||||
['repo', new Set(['view'])]
|
||||
])
|
||||
|
||||
export function classifyGhRateLimitBucket(args: readonly string[]): GhRateLimitBucket {
|
||||
const command = args[0]
|
||||
if (command === 'search') {
|
||||
return 'search'
|
||||
}
|
||||
if (command !== 'api') {
|
||||
return 'core'
|
||||
return GH_GRAPHQL_SUBCOMMANDS.get(command)?.has(args[1]) ? 'graphql' : 'core'
|
||||
}
|
||||
const endpoint = findGhApiEndpoint(args)
|
||||
if (!endpoint) {
|
||||
@@ -114,10 +124,10 @@ export function isGhLocalOnlyCommand(args: readonly string[]): boolean {
|
||||
return args[0] === 'auth' && args[1] === 'token'
|
||||
}
|
||||
|
||||
/** Recognize primary limits; secondary limits use Retry-After handling. */
|
||||
/** Recognize primary limits (GraphQL says "already exceeded"); secondary limits use Retry-After handling. */
|
||||
export function isGhPrimaryRateLimitStderr(stderr: string): boolean {
|
||||
const s = stderr.toLowerCase()
|
||||
return s.includes('api rate limit exceeded') && !s.includes('secondary rate limit')
|
||||
return /api rate limit (?:already )?exceeded/.test(s) && !s.includes('secondary rate limit')
|
||||
}
|
||||
|
||||
export function recordGhPrimaryRateLimit(
|
||||
|
||||
@@ -139,9 +139,29 @@ describe('ghExecFileAsync rate-limit breaker', () => {
|
||||
).rejects.toThrow('rate limit')
|
||||
|
||||
expect(
|
||||
getGhRateLimitBlockedUntilMs('core', Date.now(), 'native:github.acme-corp.com')
|
||||
getGhRateLimitBlockedUntilMs('graphql', Date.now(), 'native:github.acme-corp.com')
|
||||
).not.toBeNull()
|
||||
expect(getGhRateLimitBlockedUntilMs('core', Date.now(), 'native:github.com')).toBeNull()
|
||||
expect(getGhRateLimitBlockedUntilMs('graphql', Date.now(), 'native:github.com')).toBeNull()
|
||||
})
|
||||
|
||||
it('trips the graphql bucket on a GraphQL rate limit and leaves REST calls running', async () => {
|
||||
mockGhFailure('GraphQL: API rate limit already exceeded for user ID 1775218.')
|
||||
await expect(
|
||||
ghExecFileAsync(['pr', 'view', '12', '--repo', 'a/b', '--json', 'number'])
|
||||
).rejects.toThrow('rate limit')
|
||||
expect(getGhRateLimitBlockedUntilMs('graphql')).not.toBeNull()
|
||||
expect(getGhRateLimitBlockedUntilMs('core')).toBeNull()
|
||||
|
||||
await expect(
|
||||
ghExecFileAsync(['issue', 'close', '5', '--repo', 'a/b'], { idempotent: false })
|
||||
).rejects.toMatchObject({ ghRateLimitBlocked: true })
|
||||
expect(spawnMock).toHaveBeenCalledTimes(1)
|
||||
|
||||
mockGhSuccess('[]')
|
||||
await expect(ghExecFileAsync(['api', 'repos/a/b/pulls'])).resolves.toMatchObject({
|
||||
stdout: '[]'
|
||||
})
|
||||
expect(spawnMock).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('scopes a WSL UNC-cwd 403 to the distro runtime and probes that scope', async () => {
|
||||
|
||||
@@ -111,7 +111,7 @@ describe('updateIssueBySlug', () => {
|
||||
expect(runRestMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('guards and accounts state mutations in the selected host quota scope', async () => {
|
||||
it('guards and accounts state mutations in the selected host GraphQL quota', async () => {
|
||||
const args = {
|
||||
owner: 'acme',
|
||||
repo: 'widgets',
|
||||
@@ -122,8 +122,8 @@ describe('updateIssueBySlug', () => {
|
||||
|
||||
await expect(updateIssueBySlug(args)).resolves.toEqual({ ok: true })
|
||||
|
||||
expect(repositoryRateLimitGuardMock).toHaveBeenCalledWith(args, 'core')
|
||||
expect(noteRepositoryRateLimitSpendMock).toHaveBeenCalledWith(args, 'core')
|
||||
expect(repositoryRateLimitGuardMock).toHaveBeenCalledWith(args, 'graphql')
|
||||
expect(noteRepositoryRateLimitSpendMock).toHaveBeenCalledWith(args, 'graphql')
|
||||
})
|
||||
|
||||
it('does not spawn a state mutation while that host quota is blocked', async () => {
|
||||
|
||||
@@ -107,7 +107,7 @@ async function updateIssueState(args: UpdateIssueBySlugArgs): Promise<GitHubProj
|
||||
if (state === undefined) {
|
||||
return { ok: true }
|
||||
}
|
||||
const guard = repositoryRateLimitGuard(args, 'core')
|
||||
const guard = repositoryRateLimitGuard(args, 'graphql')
|
||||
if (guard.blocked) {
|
||||
return { ok: false, error: rateLimitedError(guard) }
|
||||
}
|
||||
@@ -125,7 +125,7 @@ async function updateIssueState(args: UpdateIssueBySlugArgs): Promise<GitHubProj
|
||||
}
|
||||
}
|
||||
await acquire()
|
||||
noteRepositoryRateLimitSpend(args, 'core')
|
||||
noteRepositoryRateLimitSpend(args, 'graphql')
|
||||
try {
|
||||
await ghExecFileAsync(stateArgs, {
|
||||
encoding: 'utf-8',
|
||||
|
||||
Reference in New Issue
Block a user