Testing Farm Status

Uh oh! It looks like you have disabled JavaScript. Please enable scripting to enhance your experience on this website.

Experiencing disruptions

Outage 2025-03-28 10:00 UTC - Start of planned Brew outage window. BaseOS-CI is suspended, Red Hat Ranch is affected for manually triggered brew task testing

Public Ranch Outage - bug in security groups cleanup. →

Public Ranch

Outage 2025-03-06 15:30 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits.

Update 2025-03-06 22:00 UTC - Provisioning unblocked by deleting some security groups, we will monitor this.

Core Operational

Public Ranch Disrupted

Red Hat Ranch Disrupted

BaseOS CI Disrupted

Cyborg Development Cluster Operational

Cyborg Loki Operational

Fedora CI Operational

Fedora CI / installability Operational

2025 (17)

March 28, 2025 at 10:00 AM UTC

December 18, 2024 at 9:00 AM UTC

December 10, 2023 at 12:00 AM UTC

December 6, 2022 at 8:00 AM UTC

December 29, 2021 at 11:00 AM UTC

Incident history

Planned brew outage

Red Hat Ranch Outage - provisioning constrained

Red Hat Ranch Outage - artifacts storage not accessible

Public Ranch Outage - bug in security groups cleanup.

OpenStack outage in Red Hat ranch

RHIVOS HW pool outage - several issues

Beaker Outage - several issues

Beaker Outage

Public Ranch Outage - bug in security groups cleanup.

Public Ranch Outage - expired composes

Red Hat Ranch Outage - issue with Artemis deployment

Public Ranch Outage - artifacts storage out of space

Red Hat Ranch Outage - issue with Artemis deployment

Public Ranch Outage - bug in security groups cleanup.

Red Hat Ranch Outage - issue with Artemis deployment

Fedora Rawhide outdated image

Fedora Rawhide outdated image

Fedora CI is not triggering on events due to expired Fedora message bus

Brew outage https://brewweb.engineering.redhat.com/brew/

Testing Farm fails to authenticate to OpenStack

Resilience Testing outage

artifacts.osci.redhat.com outage

RHEL-9.6.0-image-mode is broken

download.devel SSL certificate expired

changed script for postinstall broke jobs in beaker

RHEL 10 CTC outage on Red Hat Ranch - broken compose

Fedora 41 and Rawhide outage on public ranch - 0xe2 breaking ansible

Red Hat Ranch / BaseOS CI Outage - provisionig issues during the weekend

Misconfigured worker in public ranch

Testing Farm 2024-10.1 released

Testing Farm API down

Gitlab CEE outage affecting jobs run on BaseOS CI and Red Hat ranch.

Fedora 41 outage on Red Hat ranch

Fedora 41 outage on public ranch.

BaseOS CI and Red Hat Ranch affected by provisioning outages. BaseOS CI down.

BaseOS CI and Red Hat Ranch affected by OpenShift outage

BaseOS CI suspended for changes in UMB certificate.

Public Ranch Outage - no archive disk space

Public Ranch - broken Fedora repositories

Red Hat Ranch / BaseOS CI Outage - provisionig issues

Red Hat Ranch Outage - large logs consuming all the space. Testing-Farm dispatcher is disabled

Public ranch outage - workers not able to connect to Artemis.

Fedora 40 small disk size

Red Hat Ranch and BaseOS-CI Outage - dns misconfiguration, some crucial sites not accessible

Testing Farm API outage and longer queues on both ranches.

All Fedora images outdated, public ranch requests processing paused.

Fedora Rawhide outdated image

Beaker PPC64 Outage - large queue because of limited hardware

Public and Redhat Ranch Outage - broken deployment

Public Ranch Outage - artifacts out of space

Public Ranch Outage - complete outage, request will stay queued or error out

Cyborg Loki is down

Public Ranch - Large queue

Red Hat Ranch and BaseOS-CI Outage - broken krb infra

Red Hat Ranch Outage - large queue, jobs getting delayed

Red Hat Ranch Outage - large queue, jobs getting delayed

Public Ranch Outage - possible large queue due to deadlocked workers

Public Ranch Outage - artifacts storage out of space

Public Ranch Outage - F40 images expired

Public Ranch Outage - large queue, high error rate due to broken workers

Public Ranch Outage - Random issues with access to koji

Public Ranch Outage - Broken Fedora Rawhide images

Public Ranch Outage - Expired images

Public Ranch Outage - Fedora 40 expired

Red Hat Ranch Degraded

Public Ranch Outage - Expired images

Public Ranch Outage - Artemis requests in error state

Long queues in Public and Red Hat ranch

Public Ranch artifact storage ran out of disk space

Public Ranch large queue - expect delays

Public Ranch large queue - expect delays

Public Ranch large queue - expect delays

BaseOS CI suspended for urgent jenkins controller upgrade.

BaseOS CI suspended for urgent jenkins controller upgrade.

Public Ranch Outage - Artemis requests in error state

Public Ranch Outage

Public Ranch reservations are broken

Fedora CI installability test affected by systemd/selinux issue.

Testing Farm Public Ranch outage due to expired images.