Testing Farm Status

Uh oh! It looks like you have disabled JavaScript. Please enable scripting to enhance your experience on this website.

Experiencing disruptions

Public Ranch provisioning unstable, high queue →

Public Ranch Fedora CI Fedora CI / installability

Outage | 2025-11-12 08:00:00 UTC

We are seeing issues with provisioning in Public ranch. Investigation is in progress.

NVIDIA V100 GPU not available →

Public Ranch

Outage | 2025-05-05 06:00:00 UTC

Unfortunately the instances with NVIDIA V100 GPUs are not available in the availability zone we use. AWS is working on scaling up to meet the demand, but there is currently no ETA. Use the NVIDIA K80 instances if you can, if not please wait, we are working on moving our workloads to multiple AZs, but it might take a few days :( Sorry for the inconvenience.

Fedora broken on aarch64, s390x and ppc64le in Beaker →

Red Hat Ranch BaseOS CI

Outage | 2025-03-10 06:00:00 UTC

Fedora testing on s390x, ppc64le and aarch64 is broken. Please share your concern in https://redhat.service-now.com/surl.do?n=INC3438634.

Update | 2025-05-27 08:20:00 UTC

The issue is still present. We have users who are failing to conduct their testing in Fedora-Rawhide and Fedora-42.

Update | 2025-11-03 10:43:00 UTC

This issue is still present. Please update the service now help ticket.

Public Ranch Disrupted

Red Hat Ranch Disrupted

BaseOS CI Disrupted

Cyborg Development Cluster Operational

Cyborg Loki Operational

Fedora CI Disrupted

Fedora CI / installability Disrupted

2025 (101)

November 12, 2025 at 1:00 PM UTC

December 18, 2024 at 9:00 AM UTC

December 10, 2023 at 12:00 AM UTC

December 6, 2022 at 8:00 AM UTC

December 29, 2021 at 11:00 AM UTC

Incident history

Public Ranch provisioning unstable, high queue

Testing Farm updating AWS DNS setup to support DNSSEC, failover to Openstack and Beaker

Beaker pools not working

AWS world-wide outage affecting Testing Farm

quay.io is down

Provisioning unstable in Testing Farm Public Ranch

Testing Farm 2025-09.1 was deployed

Brew outage

Fedora Koji Outage

Fedora Koji Outage

Fedora CI not triggering except bodhi tests

Cyborg Loki is down, due to migration to S3

Fedora and CentOS stream compose broken due to download.bos down

BaseOS-CI issue with UMB

Wow fails on connection to TCMS in BaseOS CI

BaseOS-CI jenkins controller migration

Default instance for s390x has 10GiB disk size

BaseOS CI is unstable

Provisioning errors in Public Ranch due to an outage

Beaker provisioning issues

Testing Farm public ranch large queue

BaseOS CI failing on wow

Fedora provisioning errors

Fedora CI Jenkins is down

Red Hat Gitlab Access Limited

baseos-ci queue

Public Ranch outage

Fedora Infra outage affecting koji and dist-git

Public Ranch Outage - bug in security groups cleanup.

BaseOS-CI stuck pipelines

Compose update delayed due to gitlab workers remaining in pending state.

BaseOS CI and Testing Farm testing blocked on artifacts.osci.redhat.com outage

Public Ranch deployed to 2025-06.1 release

OpenStack outage

BaseOS CI artifacts upload not working

Storage almost full - purging

BaseOS CI auto-scaling broken, testing stuck

src.fedoraproject.org is down

src.fedoraproject.org down

BaseOS CI stability issues

Stuck artemis in redhat ranch

Red Hat Ranch and BaseOS CI not working due to provisioner outage

IT resilience testing multiple SP services down

Fedora Rawhide broken due to Python3.14 upgrade

Public ranch testing blocked

Testing Farm testing disabled on s390x and ppc64le

BaseOS-CI restart

Disabled direct beaker pipelines in BaseOS-CI

ppc64le testing blocked

RHEL modules testing broken

workflow-tomorrow tests will not run

Beaker outage

RHEL-9.6.0 and RHEL-10.0 image update images are outdated

RHEL9 and RHEL10 compose update blocked on IBM cloud outage

Planned BaseOS-CI outage - migration

Public ranch provisioning is blocked

Artemis issue with policies

Some rpminspect jobs terminated on out-of-memory

ansible-core breaks ansible-lint

Instances gone during test execution

Gitlab.com outage causing various requests to fail

rpminspect requests failing on out-of-memory

BaseOS-CI outage - jenkins migration

Testing Farm Red Hat ranch high queue

Beaker outage

Rawhide requests erroring out on Testing Farm

BaseOS-CI outage - migration

BaseOS-CI outage - migration 2nd try

Planned BaseoOS-CI outage (part of migration)

BRQ beaker lab aborting jobs

NVIDIA V100 GPU not available

Red Hat Ranch blocked on high queue

RHEL-9.6.0 not schedulable on Testing Farm

BaseOS CI break with disk space hardware requirements

Some requests cancelled by an error.

Artifacts storage outage for BaseOS CI and Testing Farm

High load in testing-farm

Beaker Outage on Red Hat ranch

Large queue on Red Hat ranch