Incident history

---

2023 (31)

September 21, 2023 at 12:00 AM UTC

BaseOS CI outage due to network issues

Resolved in under a minute

September 18, 2023 at 2:00 PM UTC

Console logs are disabled due to a discovered bug

Resolved in under a minute

September 17, 2023 at 8:00 AM UTC

Red Hat Ranch outage during weekend

▲ This issue is not resolved yet September 15, 2023 at 8:00 AM UTC

Console log has wrong link in results viewer or not available at all

▲ This issue is not resolved yet September 11, 2023 at 8:00 AM UTC

Composes not updating on Red Hat ranch

▲ This issue is not resolved yet September 6, 2023 at 8:00 AM UTC

Cyborg Loki is being migrated to S3 storage and better deployment

▲ This issue is not resolved yet August 21, 2023 at 8:00 AM UTC

Beaker testing blocked on Red Hat Ranch and BaseOS CI

Resolved after 4h 0m of downtime

August 7, 2023 at 12:00 AM UTC

Large queue times on Red Hat Ranch

Resolved after 8h 22m of downtime

July 27, 2023 at 9:00 AM UTC

Red Hat Ranch affected by large queue

Resolved after 3h 11m of downtime

July 19, 2023 at 8:00 AM UTC

Public ranch testing blocked

Resolved after 3h 0m of downtime

July 18, 2023 at 8:00 AM UTC

Fedora Rawhide testing blocked on dnf5 breakage

Resolved after 6h 0m of downtime

June 22, 2023 at 9:00 AM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 528h 0m of downtime

June 21, 2023 at 12:30 PM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 4h 30m of downtime

June 1, 2023 at 10:30 AM UTC

Red Hat Ranch affected by RH infra outage

Resolved after 21h 30m of downtime

May 29, 2023 at 2:00 AM UTC

Expired Fedora-Rawhide images in Public Ranch

Resolved after 5h 0m of downtime

May 5, 2023 at 2:00 AM UTC

Unexpected outage due to new release 2023-04.1

Resolved after 10h 0m of downtime

May 2, 2023 at 8:00 AM UTC

Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Resolved in under a minute

April 28, 2023 at 4:00 AM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m of downtime

April 18, 2023 at 2:00 PM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m of downtime

April 14, 2023 at 3:00 AM UTC

Public Ranch - expired images

Resolved in under a minute

April 14, 2023 at 12:00 AM UTC

CentOS Stream 9 aarch64 broken in Public Ranch

Resolved after 92h 0m of downtime

April 6, 2023 at 10:00 AM UTC

Red Hat Ranch - deploying 2023-03.1 release - large queue expected

Resolved after 4h 0m of downtime

April 3, 2023 at 8:00 AM UTC

Red Hat Ranch - provisioning outage

Resolved in under a minute

March 29, 2023 at 8:00 AM UTC

Red Hat Ranch - larger queue

Resolved after 60m of downtime

March 28, 2023 at 8:00 PM UTC

Public ranch - Artemis resource ended in 'error' state

Resolved after 13h 0m of downtime

March 16, 2023 at 8:00 AM UTC

CentOS Stream 8 broken in Public Ranch

Resolved after 99h 0m of downtime

March 10, 2023 at 8:00 AM UTC

Fedora-Rawhide broken in Red Hat Ranch

Resolved after 148h 0m of downtime

February 10, 2023 at 9:00 AM UTC

Red Hat internal infrastructure outage

Resolved after 33h 0m of downtime

January 30, 2023 at 4:00 PM UTC

BaseOS CI - large queue, expect delays.

Resolved after 338h 0m of downtime

January 25, 2023 at 10:00 AM UTC

Public Ranch - large queue, expect delays.

Resolved after 7h 0m of downtime

January 24, 2023 at 3:00 PM UTC

Public Ranch outage - requests stuck in queued

Resolved after 19h 0m of downtime

2022 (19)

December 6, 2022 at 8:00 AM UTC

BaseOS CI Outage

Resolved after 9h 0m of downtime

November 30, 2022 at 9:00 AM UTC

Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Resolved after 7h 30m of downtime

November 29, 2022 at 8:00 AM UTC

Red Hat Ranch outage - artifacts storage empty

Resolved after 2h 30m of downtime

November 22, 2022 at 2:00 PM UTC

Public Ranch Outage - artifacts storage out of inodes

Resolved after 6h 0m of downtime

October 31, 2022 at 8:00 AM UTC

BaseOS CI Outage - UMB message sending

Resolved after 4h 0m of downtime

October 10, 2022 at 8:00 AM UTC

Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Resolved after 30h 20m of downtime

September 22, 2022 at 6:00 AM UTC

Public Ranch outage - Copr API v2 is deprecated

Resolved after 18h 30m of downtime

August 12, 2022 at 6:00 PM UTC

BaseOS CI Outage

Resolved after 67h 0m of downtime

August 11, 2022 at 8:00 AM UTC

CentOS Stream Koji Outage

Resolved after 2h 0m of downtime

August 1, 2022 at 11:00 AM UTC

Cyborg Development Cluster outage

Resolved after 2h 0m of downtime

June 16, 2022 at 9:00 AM UTC

CentOS Stream 9 repositories broken in CI

Resolved after 6h 0m of downtime

June 2, 2022 at 8:00 AM UTC

Cyborg Development Cluster credentials outage

Resolved after 7h 0m of downtime

March 25, 2022 at 12:00 PM UTC

Covscan testing blocked on BaseOS CI

Resolved after 72h 0m of downtime

February 8, 2022 at 12:00 PM UTC

rpminspect fails with "internal libpod error"

Resolved after 1441h 0m of downtime

February 5, 2022 at 8:00 PM UTC

Fedora CI outage - AWS images are gone

Resolved after 49h 25m of downtime

January 31, 2022 at 10:00 PM UTC

Broken rpmdeplint and rpminspect in Fedora CI

Resolved after 40h 30m of downtime

January 27, 2022 at 10:30 AM UTC

Large queue times on Red Hat Ranch

Resolved after 698h 22m of downtime

January 3, 2022 at 5:00 PM UTC

Large queue times on Public Ranch

Resolved after 27h 0m of downtime

January 3, 2022 at 12:17 AM UTC

Known issues  ℹ

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

2021 (3)

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m of downtime

December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m of downtime

December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m of downtime