Please read announcement
Core   (?) Testing Farm Core Operational
Public Ranch   (?) Testing Farm Public Ranch Operational
Red Hat Ranch   (?) Testing Farm Red Hat Ranch Operational
BaseOS CI   (?) BaseOS CI system Maintenance
Cyborg Development Cluster   (?) Red Hat Cyborg Development Cluster Operational

Incident history


2023 (3)

January 30, 2023 at 4:00 PM UTC

BaseOS CI - large queue, expect delays.

◆ This issue is not resolved yet
January 25, 2023 at 10:00 AM UTC

Public Ranch - large queue, expect delays.

Resolved after 7h 0m of downtime
January 24, 2023 at 3:00 PM UTC

Public Ranch outage - requests stuck in queued

Resolved after 19h 0m of downtime

2022 (19)

December 6, 2022 at 8:00 AM UTC

BaseOS CI Outage

Resolved after 9h 0m of downtime
November 30, 2022 at 9:00 AM UTC

Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Resolved after 7h 30m of downtime
November 29, 2022 at 8:00 AM UTC

Red Hat Ranch outage - artifacts storage empty

Resolved after 2h 30m of downtime
November 22, 2022 at 2:00 PM UTC

Public Ranch Outage - artifacts storage out of inodes

Resolved after 6h 0m of downtime
October 31, 2022 at 8:00 AM UTC

BaseOS CI Outage - UMB message sending

Resolved after 4h 0m of downtime
October 10, 2022 at 8:00 AM UTC

Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Resolved after 30h 20m of downtime
September 22, 2022 at 6:00 AM UTC

Public Ranch outage - Copr API v2 is deprecated

Resolved after 18h 30m of downtime
August 12, 2022 at 6:00 PM UTC

BaseOS CI Outage

Resolved after 67h 0m of downtime
August 11, 2022 at 8:00 AM UTC

CentOS Stream Koji Outage

Resolved after 2h 0m of downtime
August 1, 2022 at 11:00 AM UTC

Cyborg Development Cluster outage

Resolved after 2h 0m of downtime
June 16, 2022 at 9:00 AM UTC

CentOS Stream 9 repositories broken in CI

Resolved after 6h 0m of downtime
June 2, 2022 at 8:00 AM UTC

Cyborg Development Cluster credentials outage

Resolved after 7h 0m of downtime
March 25, 2022 at 12:00 PM UTC

Covscan testing blocked on BaseOS CI

Resolved after 72h 0m of downtime
February 8, 2022 at 12:00 PM UTC

rpminspect fails with "internal libpod error"

Resolved after 1441h 0m of downtime
February 5, 2022 at 8:00 PM UTC

Fedora CI outage - AWS images are gone

Resolved after 49h 25m of downtime
January 31, 2022 at 10:00 PM UTC

Broken rpmdeplint and rpminspect in Fedora CI

Resolved after 40h 30m of downtime
January 27, 2022 at 10:30 AM UTC

Large queue times on Red Hat Ranch

Resolved after 698h 22m of downtime
January 3, 2022 at 5:00 PM UTC

Large queue times on Public Ranch

Resolved after 27h 0m of downtime
January 3, 2022 at 12:17 AM UTC

Known issues  ℹ

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

2021 (3)

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m of downtime
December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m of downtime
December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m of downtime