Big queue in Red Hat ranch and BaseOS CI, broken Beaker testing

Outage | 2026-07-14 14:15:00 UTC

There is a big queue of test requests in Red Hat ranch. We are scaling up workers to decrease queue time.

Update | 2026-07-14 20:46:00 UTC

Large queue is gone. We are monitoring further the situation. Jobs are progressing well.

Update | 2026-07-15 08:31:00 UTC

Due to an additional bug in our provisioner surfacing during this load, some of the beaker jobs might get stuck forever and would not get provisioned. Please restart your testing if your jobs are still running and not getting any traction.

Update | 2026-07-16 13:40:00 UTC

This is ongoing and exploding :(

Update | 2026-07-16 21:32:00 UTC

Trend was finally reverted and we are trying to bring down the queue from all time high. We expect things to get better in the morning.

Update | 2026-07-17 16:51:00 UTC

Large queue is back, we are fighting it, next update in 2 hours

Update | 2026-07-17 21:37:00 UTC

Queue is finally almost gone \o/

Resolved | 2026-07-19 23:42:00 UTC

The queue is gone. Due to an outage of the artifact storage some jobs might have failed run in the last days. Please restart your testing if you see unexpected issues.