Testing Farm Status

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m

December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m

December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m

Incident history

Provisioning errors due to upgrade to Artemis v0.0.95

RH ranch is affected by expired Brew certificates

Testing Farm fails to send artifacts to artifact server.

Public ranch high queue

Testing Farm API outage

Testing Farm Red Hat ranch testing paused - large queue

Beaker download location outage - expired SSL certificate

Testing on Beaker will error out due to account disablement on Beaker side

Unexpected Artemis outage

Fedora copr access requires now valid RH CA certs for RH users and RH hosted

Testing Farm requests against Beaker failing on Ansible

Fedora composes broken in RH ranch, use Fedora-XY-Cloud-Base variants

Large queue at Red Hat ranch

Big queue in red hat ranch

Internal Copr Outage

API outage

Testing Farm complete outage

BaseOS CI triggering is blocked on expired UMB cert

ibmcloud power provisioning broken

Failing installations from Copr repositories on Fedora

Large queue on public ranch, raising also on redhat ranch

BaseOS-CI worker outage

Beaker testing disabled on Red Hat ranch and BaseOS CI

Public ranch provisioning issues

Beaker Outage on Red Hat ranch

Fedora-eln testing broken due to koji segfaulting

Public Ranch - high error rate, large queue

Storage outage

Aborted jobs due to artifact storage outage - BaseOS CI and Testing Farm Red Hat ranch

API unstable

Unstable beaker provisioning

Big queue in Red Hat ranch and BaseOS CI, broken Beaker testing

Broken ansible guest-setup on RH ranch

RHEL-10 testing blocked on broken rcmtools repo

Testing Farm public ranch large queue

Testing Farm API outage

BaseOS CI testing blocked on tcms SSL certificate expiring

API not stable

testik

Installability tests fail at AVC check

IBMCloud Power outage

CTS outage affecting Testing Farm, expect queue

quay.io outage

Artifacts storage for RH ranch outage

Red Hat ranch dispatching blocked.

Large load on RH ranch, we are scaling up

Higher error rate on Testing Farm Red Hat ranch

Fedora Rawhide provisioning often fails due to service ordering bug

Large queue on Red Hat ranch

Composes not update in Testing Farm

Rpminpsect on Fedora-CI PRs are failing when trying to download a scratch-build

Red Hat ranch requests are broken due to Artemis

BaseOS CI is not triggering

Large queue on public ranch

Elevated API errors

Testing Farm RH ranch not processing job from April 18th

Errors when regenerating tokens via the UI

Outdated image mode

Fedora Rawhide and ELN composes provisioning errors out

Testing Farm container based testing is broken

All Beaker pools are disabled due to Beaker Scheduled Maintenance

Testing Farm 2026-03.2 being deployed

ibmcloud-ppc64le cloud disabled

Beaker's disrupting Red Hat ranch.

Koji outage affects public ranch

Testing Farm outage - Cannot execute the rule

IBM Cloud ppc64le outage

RHEL-9.9.0 and RHEL-10.3 image mode testing blocked

Testing Farm 2026-02.2 was deployed on 2026-02-19

RHEL-9.9, RHEL-10.1, RHEL-10.2 and RHEL-10.3 compose update blocked

Fedora CI jobs on bodhi updates are not triggered

Testing-farm jobs on rawhide fail signature check

Large queue on public ranch due to provisioning outage

Some requests in Public Ranch are missing artifacts

Broken Fedora repositories

RHEL-8.10-ppc64le outage due to standalone python

Fedora 44 not available in Testing Farm

High queue on public ranch, expect delays

Testing Farm 2026-02.1 deployment in progress