Testing Farm Status

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m

December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m

December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m

Incident history

Fedora-eln testing broken due to koji segfaulting

Public Ranch - high error rate, large queue

Storage outage

Aborted jobs due to artifact storage outage - BaseOS CI and Testing Farm Red Hat ranch

API unstable

Unstable beaker provisioning

Big queue in Red Hat ranch and BaseOS CI, broken Beaker testing

Broken ansible guest-setup on RH ranch

RHEL-10 testing blocked on broken rcmtools repo

Testing Farm public ranch large queue

Testing Farm API outage

BaseOS CI testing blocked on tcms SSL certificate expiring

API not stable

testik

Installability tests fail at AVC check

IBMCloud Power outage

CTS outage affecting Testing Farm, expect queue

quay.io outage

Artifacts storage for RH ranch outage

Red Hat ranch dispatching blocked.

Large load on RH ranch, we are scaling up

Higher error rate on Testing Farm Red Hat ranch

Fedora Rawhide provisioning often fails due to service ordering bug

Large queue on Red Hat ranch

Composes not update in Testing Farm

Rpminpsect on Fedora-CI PRs are failing when trying to download a scratch-build

Red Hat ranch requests are broken due to Artemis

BaseOS CI is not triggering

Large queue on public ranch

Elevated API errors

Testing Farm RH ranch not processing job from April 18th

Errors when regenerating tokens via the UI

Outdated image mode

Fedora Rawhide and ELN composes provisioning errors out

Testing Farm container based testing is broken

All Beaker pools are disabled due to Beaker Scheduled Maintenance

Testing Farm 2026-03.2 being deployed

ibmcloud-ppc64le cloud disabled

Beaker's disrupting Red Hat ranch.

Koji outage affects public ranch

Testing Farm outage - Cannot execute the rule

IBM Cloud ppc64le outage

RHEL-9.9.0 and RHEL-10.3 image mode testing blocked

Testing Farm 2026-02.2 was deployed on 2026-02-19

RHEL-9.9, RHEL-10.1, RHEL-10.2 and RHEL-10.3 compose update blocked

Fedora CI jobs on bodhi updates are not triggered

Testing-farm jobs on rawhide fail signature check

Large queue on public ranch due to provisioning outage

Some requests in Public Ranch are missing artifacts

Broken Fedora repositories

RHEL-8.10-ppc64le outage due to standalone python

Fedora 44 not available in Testing Farm

High queue on public ranch, expect delays

Testing Farm 2026-02.1 deployment in progress

Image Mode doesn't work correctly on the redhat ranch

Red Hat Ranch and BaseOS CI not working due to provisioner outage

High load - delays and timeouts

IBM Cloud outage

BaseOS CI fails to send messages to UMB

Compose update blocked due to issues in IBM cloud

Elevated API errors in Testing Farm

High koji build queue for ci channel

UMB down

Redoc documentation shows empty page

Public ranch outage

Provisioning errors due to missing s390x images

Public ranch requests testing against copr builds are broken

Beaker outage affecting provisioning

Fedora 42 is unavailable in Beaker

Public Ranch outage - jobs not finishing, high queue

Public Ranch provisioning unstable, high queue

Testing Farm updating AWS DNS setup to support DNSSEC, failover to Openstack and Beaker

Beaker pools not working

AWS world-wide outage affecting Testing Farm

quay.io is down

Provisioning unstable in Testing Farm Public Ranch

Testing Farm 2025-09.1 was deployed

Brew outage

Fedora Koji Outage