Brennan Benson
15d2e31777
ci: bound the shell-contracts apt install so a stalled mirror cannot wedge the run ( #15532 )
...
* ci: bound the shell-contracts apt install so a stalled mirror cannot wedge the run
shell contracts wedges intermittently. It is not a lock, not a prompt, and not the
PR under test - it is download throughput with no wall-clock bound.
Measured on a passing run: apt-get update fetched 11.4 MB of index in 40s, then
apt-get install fetched 8.9 MB of packages at 65 kB/s taking 2m17s, while the
shell-contract tests the job exists to run took 14s. apt applies no wall-clock
bound to a stalled mirror, so when throughput drops below that already-poor
baseline the step runs indefinitely - observed at 12+ minutes and climbing while
every other job had passed.
The job also had no timeout-minutes, so it inherited GitHub's 6h default, and an
in-progress required check holds the whole run open and blocks rerun --failed.
Bounds both layers: timeout-minutes on the job (a passing run is ~4m30s), and
Acquire timeouts plus retries in apt.conf.d so a dead mirror fails fast while a
transient blip still passes. Written to apt.conf.d rather than onto the command
lines because pr-workflow-parallelism.test.mjs parses those invocations.
Does not make the job faster; the 3m38s of download is untouched. Scoping the
index refresh to just the PPA risks installing against a stale base index and
wants its own evidence.
* ci: bound the apt commands by wall clock, not per-connection timeouts
The first attempt at this set Acquire timeouts of 30s with 3 retries. That made
the wedge worse and the job's own timeout-minutes proved it: on this PR the
install step ran 14m26s and was killed by the 15 minute bound.
The log shows why. Acquire timeouts are per-connection, so a dead mirror costs
timeout x retries x every index file: 30s x 3 across roughly ten index files is
~15 minutes, which is what was observed. The azure archive mirror returned Ign
for every suite, apt fell back to archive.ubuntu.com, and that connection then
produced zero bytes for 14m26s.
So per-connection bounds cannot bound this step; only a wall-clock bound can.
Wraps both apt invocations in `timeout`, and drops Acquire::Retries to 1 so a
dead mirror fails once instead of multiplying. The update is already tolerant by
design, so bounding it just caps what a dead mirror costs before the install runs
against whatever index exists.
Also drops DPkg::Lock::Timeout: no lock contention was ever observed in these
logs, and an option added on speculation is not worth carrying.
2026-08-19 15:10:48 -07:00
..
2026-08-03 20:45:23 -07:00
2026-08-18 02:12:21 -07:00
2026-08-04 15:54:17 -07:00
2026-08-04 15:54:17 -07:00
2026-08-10 00:29:26 -07:00
2026-07-26 13:20:23 -07:00
2026-07-28 00:21:13 -07:00
2026-08-11 01:27:02 -07:00
2026-08-11 01:27:02 -07:00
2026-06-22 19:13:32 -07:00
2026-07-12 23:43:07 -07:00
2026-06-09 15:59:17 -07:00
2026-07-27 17:17:03 -07:00
2026-06-29 23:37:30 -07:00
2026-08-16 12:49:02 -07:00
2026-07-27 16:36:39 -07:00
2026-07-27 16:36:39 -07:00
2026-08-16 12:49:02 -07:00
2026-08-11 18:19:43 -07:00
2026-08-10 15:52:59 -07:00
2026-08-11 18:19:43 -07:00
2026-08-04 02:03:40 -07:00
2026-07-28 01:30:52 -07:00
2026-07-28 01:30:52 -07:00
2026-06-29 23:37:30 -07:00
2026-07-06 17:12:27 -07:00
2026-07-06 17:12:27 -07:00
2026-07-27 18:58:36 -07:00
2026-07-03 22:32:11 -07:00
2026-07-03 19:58:07 -07:00
2026-08-09 14:06:25 -07:00
2026-08-13 12:38:08 -07:00
2026-08-13 12:38:08 -07:00
2026-07-23 17:44:43 -07:00
2026-07-27 20:54:02 -07:00
2026-07-27 20:54:02 -07:00
2026-07-27 17:16:01 -07:00
2026-06-09 15:59:17 -07:00
2026-07-20 14:34:53 -07:00
2026-07-20 14:34:53 -07:00
2026-07-20 14:34:53 -07:00
2026-07-26 20:20:12 -07:00
2026-06-22 19:13:32 -07:00
2026-08-05 12:09:24 -07:00
2026-08-11 18:19:43 -07:00
2026-08-15 00:41:45 -07:00
2026-08-18 01:18:56 -07:00
2026-06-07 17:49:01 -07:00
2026-06-07 17:49:01 -07:00
2026-08-16 20:57:11 -07:00
2026-07-03 20:53:16 -07:00
2026-07-03 20:53:16 -07:00
2026-08-07 22:33:50 -07:00
2026-08-07 22:33:50 -07:00
2026-08-16 01:52:58 -07:00
2026-08-16 01:52:58 -07:00
2026-08-16 01:57:01 -07:00
2026-08-03 18:02:45 -07:00
2026-08-03 18:02:45 -07:00
2026-07-27 12:31:37 -07:00
2026-07-27 12:31:37 -07:00
2026-08-18 13:08:04 -07:00
2026-08-18 13:08:04 -07:00
2026-08-17 22:34:04 -07:00
2026-08-17 22:34:04 -07:00
2026-06-02 11:49:00 -07:00
2026-08-18 16:57:35 -07:00
2026-08-09 19:01:35 -07:00
2026-07-27 13:32:29 -07:00
2026-07-27 13:32:29 -07:00
2026-08-17 22:34:04 -07:00
2026-08-11 18:19:43 -07:00
2026-07-19 21:59:22 -04:00
2026-08-14 19:04:54 -04:00
2026-06-05 01:15:54 -07:00
2026-07-22 11:43:01 -07:00
2026-08-13 14:18:57 -07:00
2026-08-11 18:19:43 -07:00
2026-07-30 13:20:47 -07:00
2026-07-10 17:27:47 -07:00
2026-07-10 17:27:47 -07:00
2026-07-22 23:10:19 -07:00
2026-08-17 19:20:13 -07:00
2026-07-26 20:31:52 -07:00
2026-07-27 18:58:36 -07:00
2026-07-27 18:58:36 -07:00
2026-07-30 19:33:05 -07:00
2026-07-30 19:33:05 -07:00
2026-07-30 19:33:05 -07:00
2026-08-03 23:18:15 -07:00
2026-08-03 23:18:15 -07:00
2026-08-13 15:45:00 -07:00
2026-06-28 21:16:02 -07:00
2026-08-04 15:54:17 -07:00
2026-08-04 15:54:17 -07:00
2026-07-27 17:09:31 -07:00
2026-08-12 00:37:48 -07:00
2026-08-12 00:37:48 -07:00
2026-08-12 00:37:48 -07:00
2026-08-12 00:37:48 -07:00
2026-06-15 12:56:22 -07:00
2026-06-29 23:37:30 -07:00
2026-08-12 14:40:39 -07:00
2026-08-12 14:40:39 -07:00
2026-06-03 23:42:33 -07:00
2026-06-04 16:11:51 -04:00
2026-08-11 18:19:43 -07:00
2026-06-03 12:48:00 -07:00
2026-08-16 13:45:54 -07:00
2026-06-28 19:43:18 -07:00
2026-07-27 20:54:02 -07:00
2026-06-28 19:43:18 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-03 02:18:05 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 22:24:09 -07:00
2026-07-24 00:26:39 -07:00
2026-08-13 02:56:43 -07:00
2026-08-04 03:56:33 -07:00
2026-08-04 03:56:33 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-13 02:56:43 -07:00
2026-08-13 02:56:43 -07:00
2026-06-29 23:37:30 -07:00
2026-06-19 15:52:05 -07:00
2026-06-09 15:59:17 -07:00
2026-06-09 15:59:17 -07:00
2026-08-13 23:47:20 -07:00
2026-07-31 15:13:41 -07:00
2026-08-04 03:56:33 -07:00
2026-08-04 03:56:33 -07:00
2026-06-17 15:14:35 -07:00
2026-08-12 18:53:17 -07:00
2026-08-09 15:40:48 -07:00
2026-08-04 03:56:33 -07:00
2026-08-12 18:53:17 -07:00
2026-08-04 03:56:33 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-09 15:40:48 -07:00
2026-08-12 18:53:17 -07:00
2026-08-07 22:08:22 -07:00
2026-08-07 22:08:22 -07:00
2026-08-12 18:53:17 -07:00
2026-08-12 18:53:17 -07:00
2026-07-29 17:44:41 -07:00
2026-07-10 19:08:12 -07:00
2026-07-27 16:36:39 -07:00
2026-07-27 16:36:39 -07:00
2026-08-11 18:19:43 -07:00
2026-07-30 19:24:34 -07:00
2026-07-30 19:24:34 -07:00
2026-07-30 19:24:34 -07:00
2026-07-30 19:24:34 -07:00
2026-07-30 19:24:34 -07:00
2026-06-02 21:15:00 -07:00
2026-07-28 15:34:52 -07:00
2026-06-28 21:19:10 -07:00
2026-08-11 18:19:43 -07:00
2026-07-26 20:31:55 -07:00
2026-07-27 18:58:36 -07:00
2026-08-17 12:02:47 -07:00
2026-06-22 01:07:16 -07:00
2026-06-22 01:07:16 -07:00
2026-07-27 20:54:02 -07:00
2026-08-09 13:19:08 -07:00
2026-05-15 05:44:25 -04:00
2026-07-27 12:31:37 -07:00
2026-07-27 12:31:37 -07:00
2026-07-22 11:43:01 -07:00
2026-08-13 12:00:04 -07:00
2026-07-28 00:21:13 -07:00
2026-08-16 12:49:02 -07:00
2026-07-30 19:33:05 -07:00
2026-08-11 12:20:20 -07:00
2026-08-17 23:28:38 -07:00
2026-08-18 14:09:12 -07:00
2026-07-27 16:26:43 -07:00
2026-08-19 15:10:48 -07:00
2026-07-27 13:32:29 -07:00
2026-07-27 13:32:29 -07:00
2026-05-30 13:09:17 -07:00
2026-07-27 20:54:02 -07:00
2026-05-30 21:46:13 -07:00
2026-07-28 00:21:13 -07:00
2026-07-22 00:22:36 -07:00
2026-07-22 00:22:36 -07:00
2026-08-17 21:31:20 -07:00
2026-08-17 21:31:20 -07:00
2026-08-10 01:17:14 -07:00
2026-07-27 16:40:59 -07:00
2026-08-11 18:19:43 -07:00
2026-07-30 18:08:30 -07:00
2026-08-17 23:28:38 -07:00
2026-06-30 16:36:59 -07:00
2026-07-25 03:50:25 -07:00
2026-07-25 03:50:25 -07:00
2026-08-04 15:54:17 -07:00
2026-07-23 20:20:00 -07:00
2026-07-23 20:20:00 -07:00
2026-07-23 20:20:00 -07:00
2026-07-23 20:20:00 -07:00
2026-07-23 20:20:00 -07:00
2026-08-13 22:48:24 -07:00
2026-07-28 00:21:13 -07:00
2026-07-12 23:43:07 -07:00
2026-07-26 13:19:23 -07:00
2026-07-02 18:41:23 -04:00
2026-07-27 15:48:39 -07:00
2026-07-27 15:48:39 -07:00
2026-07-27 17:18:31 -07:00
2026-08-10 15:52:59 -07:00
2026-08-05 12:57:02 -07:00
2026-08-05 12:57:02 -07:00
2026-06-22 01:07:16 -07:00
2026-08-18 13:08:04 -07:00
2026-07-27 13:32:29 -07:00
2026-08-14 19:04:54 -04:00
2026-07-22 18:52:37 -07:00
2026-08-13 15:45:00 -07:00
2026-08-12 00:37:48 -07:00
2026-08-02 17:40:58 -07:00
2026-05-16 17:44:42 -04:00
2026-08-16 13:45:54 -07:00
2026-08-16 13:45:54 -07:00
2026-07-20 21:36:15 -07:00
2026-08-01 00:25:35 -07:00
2026-07-22 18:25:05 -07:00
2026-06-30 16:36:59 -07:00
2026-06-22 16:42:20 -07:00
2026-08-03 02:18:05 -07:00
2026-06-07 04:15:26 -07:00
2026-08-12 22:33:19 -07:00
2026-07-12 23:50:00 -07:00
2026-08-11 18:19:43 -07:00
2026-08-08 19:06:50 -07:00
2026-06-07 11:01:02 -07:00
2026-07-10 17:27:47 -07:00
2026-07-10 17:27:47 -07:00
2026-08-17 19:20:13 -07:00
2026-06-22 01:07:16 -07:00
2026-07-12 18:15:15 -07:00
2026-07-15 15:23:35 -07:00
2026-08-05 12:57:02 -07:00
2026-08-09 19:01:35 -07:00
2026-08-09 19:01:35 -07:00
2026-08-09 19:01:35 -07:00
2026-08-16 13:45:54 -07:00
2026-08-17 19:20:13 -07:00
2026-08-07 22:54:42 -07:00
2026-07-31 04:58:15 -07:00
2026-07-30 19:33:05 -07:00
2026-07-26 22:26:49 -07:00
2026-07-27 01:02:48 -07:00
2026-07-10 17:27:47 -07:00
2026-06-22 19:13:32 -07:00
2026-07-27 17:38:54 -07:00
2026-07-27 17:38:54 -07:00
2026-07-27 23:28:05 -07:00
2026-08-11 18:19:43 -07:00
2026-08-08 19:06:50 -07:00
2026-07-27 23:28:05 -07:00
2026-06-22 19:13:32 -07:00
2026-06-22 19:13:32 -07:00
2026-07-10 17:27:47 -07:00
2026-07-26 15:32:33 -07:00
2026-08-06 00:23:57 -07:00
2026-07-27 17:15:54 -07:00
2026-06-29 23:37:30 -07:00
2026-06-29 23:37:30 -07:00
2026-06-29 23:37:30 -07:00
2026-08-16 21:58:28 -07:00
2026-06-28 22:58:27 -07:00
2026-07-23 18:35:31 -07:00
2026-08-11 18:19:43 -07:00
2026-07-22 19:11:44 -07:00
2026-07-22 19:11:44 -07:00
2026-08-08 19:06:50 -07:00
2026-08-04 03:56:33 -07:00
2026-08-04 03:56:33 -07:00
2026-08-01 00:25:35 -07:00
2026-07-29 17:44:41 -07:00
2026-06-29 23:37:30 -07:00
2026-04-09 12:28:52 -07:00
2026-07-09 16:09:09 -07:00
2026-07-09 16:09:09 -07:00
2026-07-27 01:14:33 -07:00
2026-07-27 01:14:33 -07:00
2026-07-05 23:56:37 -07:00
2026-06-22 01:07:16 -07:00
2026-07-16 14:47:28 -07:00
2026-07-30 18:27:16 -07:00
2026-07-30 18:27:16 -07:00
2026-07-27 17:38:54 -07:00
2026-06-29 22:38:29 -07:00
2026-07-02 12:11:58 -07:00
2026-07-02 12:11:58 -07:00
2026-08-01 00:25:35 -07:00
2026-07-19 20:25:52 -07:00
2026-07-27 15:48:39 -07:00
2026-07-31 06:08:26 -07:00
2026-08-10 15:52:59 -07:00
2026-08-09 22:24:33 -07:00
2026-08-13 20:44:16 -07:00
2026-06-09 10:43:04 -07:00
2026-08-18 02:12:21 -07:00
2026-08-08 14:06:34 -07:00
2026-08-09 12:53:13 -07:00
2026-07-27 18:58:36 -07:00