Incident history

---

2025 (29)

April 23, 2025 at 10:00 PM UTC

Red Hat Ranch blocked on high queue

Resolved after 47h 4m

April 17, 2025 at 3:24 PM UTC

RHEL-9.6.0 not schedulable on Testing Farm

Resolved after 2h 36m

April 17, 2025 at 8:00 AM UTC

BaseOS CI break with disk space hardware requirements

Resolved after 24h 0m

April 14, 2025 at 9:52 AM UTC

Some requests cancelled by an error.

Resolved after 8m

April 11, 2025 at 1:00 PM UTC

Artifacts storage outage for BaseOS CI and Testing Farm

Resolved after 2h 10m

April 9, 2025 at 4:00 PM UTC

High load in testing-farm

Resolved after 3h 26m

April 8, 2025 at 10:00 AM UTC

Beaker Outage on Red Hat ranch

Resolved after 143h 50m

April 7, 2025 at 4:02 PM UTC

Large queue on Red Hat ranch

Resolved after 4h 38m

April 7, 2025 at 7:58 AM UTC

RHEL-10.0 is expired on Red Hat Ranch

Resolved after 3h 13m

April 3, 2025 at 7:30 AM UTC

Testing Farm outage due to deployment

Resolved after 1h 30m

April 3, 2025 at 4:00 AM UTC

Red Hat Ranch - expect delays for RHEL-9.5.0 and RHEL-10.0

Resolved after 12h 0m

April 1, 2025 at 2:00 PM UTC

Public Ranch Outage - outdated composes

Resolved after 47m

March 28, 2025 at 10:00 AM UTC

Planned brew outage

Resolved after 26h 0m

March 17, 2025 at 9:00 AM UTC

Red Hat Ranch Outage - provisioning constrained

Resolved after 23h 0m

March 7, 2025 at 10:00 AM UTC

Red Hat Ranch Outage - artifacts storage not accessible

Resolved after 6h 0m

March 6, 2025 at 3:30 PM UTC

Public Ranch Outage - bug in security groups cleanup.

Resolved in under a minute

February 27, 2025 at 12:00 PM UTC

OpenStack outage in Red Hat ranch

Resolved after 6h 0m

February 24, 2025 at 8:28 AM UTC

RHIVOS HW pool outage - several issues

Resolved after 1h 58m

February 12, 2025 at 8:00 AM UTC

Beaker Outage - several issues

Resolved after 169h 0m

February 11, 2025 at 2:00 PM UTC

Beaker Outage

Resolved after 4h 0m

February 7, 2025 at 2:00 PM UTC

Public Ranch Outage - bug in security groups cleanup.

Resolved after 2h 0m

February 2, 2025 at 4:00 AM UTC

Public Ranch Outage - expired composes

Resolved after 28h 0m

January 31, 2025 at 6:00 PM UTC

Red Hat Ranch Outage - issue with Artemis deployment

Resolved after 56h 0m

January 29, 2025 at 4:00 PM UTC

Public Ranch Outage - artifacts storage out of space

Resolved after 2h 0m

January 17, 2025 at 9:00 AM UTC

Red Hat Ranch Outage - issue with Artemis deployment

Resolved after 2h 0m

January 16, 2025 at 3:20 PM UTC

Public Ranch Outage - bug in security groups cleanup.

Resolved after 50m

January 16, 2025 at 12:20 AM UTC

Red Hat Ranch Outage - issue with Artemis deployment

Resolved after 4h 0m

January 14, 2025 at 8:00 AM UTC

Fedora Rawhide outdated image

Resolved after 26h 0m

January 9, 2025 at 8:00 AM UTC

Fedora Rawhide outdated image

Resolved after 4h 0m

2024 (57)

December 18, 2024 at 9:00 AM UTC

Fedora CI is not triggering on events due to expired Fedora message bus

Resolved after 6h 0m

December 10, 2024 at 9:00 AM UTC

Brew outage https://brewweb.engineering.redhat.com/brew/

Resolved after 220h 45m

December 10, 2024 at 8:00 AM UTC

Testing Farm fails to authenticate to OpenStack

Resolved after 7h 0m

December 6, 2024 at 12:30 PM UTC

Resilience Testing outage

Resolved after 6h 30m

December 5, 2024 at 1:00 PM UTC

artifacts.osci.redhat.com outage

Resolved after 2h 30m

November 26, 2024 at 12:00 AM UTC

RHEL-9.6.0-image-mode is broken

Resolved after 231h 30m

November 24, 2024 at 9:00 PM UTC

download.devel SSL certificate expired

Resolved after 13h 30m

November 19, 2024 at 5:00 PM UTC

changed script for postinstall broke jobs in beaker

Resolved after 38h 30m

November 19, 2024 at 12:00 AM UTC

RHEL 10 CTC outage on Red Hat Ranch - broken compose

Resolved after 362h 0m

November 15, 2024 at 12:00 AM UTC

Fedora 41 and Rawhide outage on public ranch - 0xe2 breaking ansible

Resolved after 95h 15m

November 9, 2024 at 12:00 AM UTC

Red Hat Ranch / BaseOS CI Outage - provisionig issues during the weekend

Resolved after 58h 0m

November 8, 2024 at 8:00 AM UTC

Misconfigured worker in public ranch

Resolved after 7h 0m

November 7, 2024 at 8:00 AM UTC

Testing Farm 2024-10.1 released

Resolved after 96h 0m

November 6, 2024 at 2:30 PM UTC

Testing Farm API down

Resolved after 45m

November 4, 2024 at 6:00 PM UTC

Gitlab CEE outage affecting jobs run on BaseOS CI and Red Hat ranch.

Resolved after 9h 0m

November 4, 2024 at 6:00 PM UTC

Fedora 41 outage on Red Hat ranch

Resolved after 18h 0m

November 1, 2024 at 12:00 AM UTC

Fedora 41 outage on public ranch.

Resolved after 15h 0m

November 1, 2024 at 12:00 AM UTC

BaseOS CI and Red Hat Ranch affected by provisioning outages. BaseOS CI down.

Resolved after 9h 0m

October 31, 2024 at 12:00 PM UTC

BaseOS CI and Red Hat Ranch affected by OpenShift outage

Resolved after 12h 0m

October 15, 2024 at 5:00 PM UTC

BaseOS CI suspended for changes in UMB certificate.

Resolved after 6h 0m

October 1, 2024 at 10:00 AM UTC

Public Ranch Outage - no archive disk space

Resolved after 60m

September 16, 2024 at 12:00 AM UTC

Public Ranch - broken Fedora repositories

Resolved after 19h 0m

September 3, 2024 at 8:00 PM UTC

Red Hat Ranch / BaseOS CI Outage - provisionig issues

Resolved after 17h 30m

August 30, 2024 at 8:30 AM UTC

Red Hat Ranch Outage - large logs consuming all the space. Testing-Farm dispatcher is disabled

Resolved after 5h 0m

August 28, 2024 at 10:29 AM UTC

Public ranch outage - workers not able to connect to Artemis.

Resolved after 3h 52m

August 20, 2024 at 6:00 PM UTC

Fedora 40 small disk size

Resolved after 206h 0m

August 13, 2024 at 3:40 PM UTC

Red Hat Ranch and BaseOS-CI Outage - dns misconfiguration, some crucial sites not accessible

Resolved after 9h 10m

August 8, 2024 at 4:00 AM UTC

Testing Farm API outage and longer queues on both ranches.

Resolved after 2h 15m

August 5, 2024 at 4:00 AM UTC

All Fedora images outdated, public ranch requests processing paused.

Resolved after 9h 0m

August 2, 2024 at 8:00 AM UTC

Fedora Rawhide outdated image

Resolved after 60m

July 26, 2024 at 12:00 AM UTC

Beaker PPC64 Outage - large queue because of limited hardware

Resolved after 168h 0m

July 25, 2024 at 1:00 PM UTC

Public and Redhat Ranch Outage - broken deployment

Resolved after 6h 45m

July 23, 2024 at 6:00 AM UTC

Public Ranch Outage - artifacts out of space

Resolved after 4h 0m

July 22, 2024 at 12:16 AM UTC

Public Ranch Outage - complete outage, request will stay queued or error out

Resolved after 9h 1m

July 16, 2024 at 12:00 AM UTC

Cyborg Loki is down

Resolved after 11h 30m

June 17, 2024 at 1:00 PM UTC

Public Ranch - Large queue

Resolved after 357h 0m

May 28, 2024 at 6:00 PM UTC

Red Hat Ranch and BaseOS-CI Outage - broken krb infra

Resolved after 16h 0m

May 27, 2024 at 12:00 AM UTC

Red Hat Ranch Outage - large queue, jobs getting delayed

Resolved after 12h 0m

May 19, 2024 at 6:00 AM UTC

Red Hat Ranch Outage - large queue, jobs getting delayed

Resolved after 14h 0m

May 15, 2024 at 8:00 PM UTC

Public Ranch Outage - possible large queue due to deadlocked workers

Resolved after 23h 30m

May 9, 2024 at 4:00 PM UTC

Public Ranch Outage - artifacts storage out of space

Resolved after 3h 30m

May 6, 2024 at 12:00 AM UTC

Public Ranch Outage - F40 images expired

Resolved after 7h 30m

April 30, 2024 at 6:00 PM UTC

Public Ranch Outage - large queue, high error rate due to broken workers

Resolved after 3h 30m

April 25, 2024 at 12:00 AM UTC

Public Ranch Outage - Random issues with access to koji

Resolved after 271h 15m

April 25, 2024 at 12:00 AM UTC

Public Ranch Outage - Broken Fedora Rawhide images

Resolved after 11h 0m

April 15, 2024 at 12:00 AM UTC

Public Ranch Outage - Expired images

Resolved after 8h 30m

March 29, 2024 at 12:00 AM UTC

Public Ranch Outage - Fedora 40 expired

Resolved after 17h 0m

March 28, 2024 at 12:00 AM UTC

Red Hat Ranch Degraded

Resolved after 22h 30m

March 28, 2024 at 12:00 AM UTC

Public Ranch Outage - Expired images

Resolved after 9h 30m

March 19, 2024 at 12:00 AM UTC

Public Ranch Outage - Artemis requests in error state

Resolved after 11h 0m

March 12, 2024 at 3:00 PM UTC

Long queues in Public and Red Hat ranch

Resolved after 29h 0m

March 8, 2024 at 8:00 AM UTC

Public Ranch artifact storage ran out of disk space

Resolved after 5h 0m

March 7, 2024 at 10:00 AM UTC

Public Ranch large queue - expect delays

Resolved after 6h 0m

March 5, 2024 at 3:00 PM UTC

Public Ranch large queue - expect delays

Resolved after 6h 0m

February 15, 2024 at 5:00 PM UTC

Public Ranch large queue - expect delays

Resolved in under a minute

February 2, 2024 at 8:00 AM UTC

BaseOS CI suspended for urgent jenkins controller upgrade.

Resolved after 60m

January 3, 2024 at 10:47 AM UTC

BaseOS CI suspended for urgent jenkins controller upgrade.

Resolved after 1h 43m

2023 (41)

December 10, 2023 at 12:00 AM UTC

Public Ranch Outage - Artemis requests in error state

Resolved after 35h 0m

December 2, 2023 at 12:00 AM UTC

Public Ranch Outage

Resolved after 3h 0m

November 29, 2023 at 12:00 AM UTC

Public Ranch reservations are broken

Resolved after 202h 0m

November 23, 2023 at 12:00 AM UTC

Fedora CI installability test affected by systemd/selinux issue.

Resolved after 1882h 0m

November 20, 2023 at 12:00 AM UTC

Testing Farm Public Ranch outage due to expired images.

Resolved after 10h 0m

November 19, 2023 at 6:00 AM UTC

Testing Farm Red Hat Ranch outage during the weekend

Resolved after 27h 0m

November 12, 2023 at 12:00 AM UTC

BaseOS CI limited due to outage due to BRQ network maintanance.

Resolved after 36h 0m

November 7, 2023 at 3:00 PM UTC

BaseOS CI and Testing Farm Red Hat Ranch outage due to Beaker unavailability

Resolved after 4h 0m

September 30, 2023 at 4:00 AM UTC

Expired images on Public ranch.

Resolved after 48h 0m

September 25, 2023 at 12:00 AM UTC

Testing Farm Red Hat ranch and BaseOS CI outage for RHEL9 and RHEL-8.1.0

Resolved after 16h 30m

September 21, 2023 at 12:00 AM UTC

BaseOS CI outage due to network issues

Resolved in under a minute

September 18, 2023 at 2:00 PM UTC

Console logs are disabled due to a discovered bug

Resolved in under a minute

September 17, 2023 at 8:00 AM UTC

Red Hat Ranch outage during weekend

Resolved after 8h 0m

September 15, 2023 at 8:00 AM UTC

Console log has wrong link in results viewer or not available at all.

Resolved after 274h 0m

September 11, 2023 at 8:00 AM UTC

Composes not updating on Red Hat ranch

Resolved in under a minute

September 6, 2023 at 8:00 AM UTC

Cyborg Loki is being migrated to S3 storage and better deployment

Resolved after 2788h 0m

August 21, 2023 at 8:00 AM UTC

Beaker testing blocked on Red Hat Ranch and BaseOS CI

Resolved after 4h 0m

August 7, 2023 at 12:00 AM UTC

Large queue times on Red Hat Ranch

Resolved after 8h 22m

July 27, 2023 at 9:00 AM UTC

Red Hat Ranch affected by large queue

Resolved after 3h 11m

July 19, 2023 at 8:00 AM UTC

Public ranch testing blocked

Resolved after 3h 0m

July 18, 2023 at 8:00 AM UTC

Fedora Rawhide testing blocked on dnf5 breakage

Resolved after 6h 0m

June 22, 2023 at 9:00 AM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 528h 0m

June 21, 2023 at 12:30 PM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 4h 30m

June 1, 2023 at 10:30 AM UTC

Red Hat Ranch affected by RH infra outage

Resolved after 21h 30m

May 29, 2023 at 2:00 AM UTC

Expired Fedora-Rawhide images in Public Ranch

Resolved after 5h 0m

May 5, 2023 at 2:00 AM UTC

Unexpected outage due to new release 2023-04.1

Resolved after 10h 0m

May 2, 2023 at 8:00 AM UTC

Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Resolved in under a minute

April 28, 2023 at 4:00 AM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m

April 18, 2023 at 2:00 PM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m

April 14, 2023 at 3:00 AM UTC

Public Ranch - expired images

Resolved in under a minute

April 14, 2023 at 12:00 AM UTC

CentOS Stream 9 aarch64 broken in Public Ranch

Resolved after 92h 0m

April 6, 2023 at 10:00 AM UTC

Red Hat Ranch - deploying 2023-03.1 release - large queue expected

Resolved after 4h 0m

April 3, 2023 at 8:00 AM UTC

Red Hat Ranch - provisioning outage

Resolved in under a minute

March 29, 2023 at 8:00 AM UTC

Red Hat Ranch - larger queue

Resolved after 60m

March 28, 2023 at 8:00 PM UTC

Public ranch - Artemis resource ended in 'error' state

Resolved after 13h 0m

March 16, 2023 at 8:00 AM UTC

CentOS Stream 8 broken in Public Ranch

Resolved after 99h 0m

March 10, 2023 at 8:00 AM UTC

Fedora-Rawhide broken in Red Hat Ranch

Resolved after 148h 0m

February 10, 2023 at 9:00 AM UTC

Red Hat internal infrastructure outage

Resolved after 33h 0m

January 30, 2023 at 4:00 PM UTC

BaseOS CI - large queue, expect delays.

Resolved after 338h 0m

January 25, 2023 at 10:00 AM UTC

Public Ranch - large queue, expect delays.

Resolved after 7h 0m

January 24, 2023 at 3:00 PM UTC

Public Ranch outage - requests stuck in queued

Resolved after 19h 0m

2022 (19)

December 6, 2022 at 8:00 AM UTC

BaseOS CI Outage

Resolved after 9h 0m

November 30, 2022 at 9:00 AM UTC

Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Resolved after 7h 30m

November 29, 2022 at 8:00 AM UTC

Red Hat Ranch outage - artifacts storage empty

Resolved after 2h 30m

November 22, 2022 at 2:00 PM UTC

Public Ranch Outage - artifacts storage out of inodes

Resolved after 6h 0m

October 31, 2022 at 8:00 AM UTC

BaseOS CI Outage - UMB message sending

Resolved after 4h 0m

October 10, 2022 at 8:00 AM UTC

Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Resolved after 30h 20m

September 22, 2022 at 6:00 AM UTC

Public Ranch outage - Copr API v2 is deprecated

Resolved after 18h 30m

August 12, 2022 at 6:00 PM UTC

BaseOS CI Outage

Resolved after 67h 0m

August 11, 2022 at 8:00 AM UTC

CentOS Stream Koji Outage

Resolved after 2h 0m

August 1, 2022 at 11:00 AM UTC

Cyborg Development Cluster outage

Resolved after 2h 0m

June 16, 2022 at 9:00 AM UTC

CentOS Stream 9 repositories broken in CI

Resolved after 6h 0m

June 2, 2022 at 8:00 AM UTC

Cyborg Development Cluster credentials outage

Resolved after 7h 0m

March 25, 2022 at 12:00 PM UTC

Covscan testing blocked on BaseOS CI

Resolved after 72h 0m

February 8, 2022 at 12:00 PM UTC

rpminspect fails with "internal libpod error"

Resolved after 1441h 0m

February 5, 2022 at 8:00 PM UTC

Fedora CI outage - AWS images are gone

Resolved after 49h 25m

January 31, 2022 at 10:00 PM UTC

Broken rpmdeplint and rpminspect in Fedora CI

Resolved after 40h 30m

January 27, 2022 at 10:30 AM UTC

Large queue times on Red Hat Ranch

Resolved after 698h 22m

January 3, 2022 at 5:00 PM UTC

Large queue times on Public Ranch

Resolved after 27h 0m

January 3, 2022 at 12:17 AM UTC

Known issues  ℹ

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

2021 (3)

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m

December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m

December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m