Experiencing disruptions

Public Ranch Outage - bug in security groups cleanup. →

Public Ranch

Outage 2025-03-06 15:30 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits.

Update 2025-03-06 22:00 UTC - Provisioning unblocked by deleting some security groups, we will monitor this.

Core   (?) Testing Farm Core Operational
Public Ranch   (?) Testing Farm Public Ranch Disrupted
Red Hat Ranch   (?) Testing Farm Red Hat Ranch Operational
BaseOS CI   (?) BaseOS CI system Operational
Cyborg Development Cluster   (?) Red Hat Cyborg Development Cluster Operational
Cyborg Loki   (?) Red Hat Cyborg Loki Deployment Operational
Fedora CI   (?) Affecting all Fedora CI tests Operational
Fedora CI / installability   (?) Affecting Fedora CI installability test Operational

Incident history

2025 (18)

April 1, 2025 at 2:00 PM UTC

Public Ranch Outage - outdated composes

Resolved after 47m
March 28, 2025 at 10:00 AM UTC

Planned brew outage

Resolved after 26h 0m
March 17, 2025 at 9:00 AM UTC

Red Hat Ranch Outage - provisioning constrained

Resolved after 23h 0m
March 7, 2025 at 10:00 AM UTC

Red Hat Ranch Outage - artifacts storage not accessible

Resolved after 6h 0m
March 6, 2025 at 3:30 PM UTC

Public Ranch Outage - bug in security groups cleanup.

▲ This issue is not resolved yet February 27, 2025 at 12:00 PM UTC

OpenStack outage in Red Hat ranch

Resolved after 6h 0m
February 24, 2025 at 8:28 AM UTC

RHIVOS HW pool outage - several issues

Resolved after 1h 58m
February 12, 2025 at 8:00 AM UTC

Beaker Outage - several issues

Resolved after 169h 0m
February 11, 2025 at 2:00 PM UTC

Beaker Outage

Resolved after 4h 0m
February 7, 2025 at 2:00 PM UTC

Public Ranch Outage - bug in security groups cleanup.

Resolved after 2h 0m
February 2, 2025 at 4:00 AM UTC

Public Ranch Outage - expired composes

Resolved after 28h 0m
January 31, 2025 at 6:00 PM UTC

Red Hat Ranch Outage - issue with Artemis deployment

Resolved after 56h 0m
January 29, 2025 at 4:00 PM UTC

Public Ranch Outage - artifacts storage out of space

Resolved after 2h 0m
January 17, 2025 at 9:00 AM UTC

Red Hat Ranch Outage - issue with Artemis deployment

Resolved after 2h 0m
January 16, 2025 at 3:20 PM UTC

Public Ranch Outage - bug in security groups cleanup.

Resolved after 50m
January 16, 2025 at 12:20 AM UTC

Red Hat Ranch Outage - issue with Artemis deployment

Resolved after 4h 0m
January 14, 2025 at 8:00 AM UTC

Fedora Rawhide outdated image

Resolved after 26h 0m
January 9, 2025 at 8:00 AM UTC

Fedora Rawhide outdated image

Resolved after 4h 0m

2024 (57)

December 18, 2024 at 9:00 AM UTC

Fedora CI is not triggering on events due to expired Fedora message bus

Resolved after 6h 0m
December 10, 2024 at 9:00 AM UTC

Brew outage https://brewweb.engineering.redhat.com/brew/

Resolved after 220h 45m
December 10, 2024 at 8:00 AM UTC

Testing Farm fails to authenticate to OpenStack

Resolved after 7h 0m
December 6, 2024 at 12:30 PM UTC

Resilience Testing outage

Resolved after 6h 30m
December 5, 2024 at 1:00 PM UTC

artifacts.osci.redhat.com outage

Resolved after 2h 30m
November 26, 2024 at 12:00 AM UTC

RHEL-9.6.0-image-mode is broken

Resolved after 231h 30m
November 24, 2024 at 9:00 PM UTC

download.devel SSL certificate expired

Resolved after 13h 30m
November 19, 2024 at 5:00 PM UTC

changed script for postinstall broke jobs in beaker

Resolved after 38h 30m
November 19, 2024 at 12:00 AM UTC

RHEL 10 CTC outage on Red Hat Ranch - broken compose

Resolved after 362h 0m
November 15, 2024 at 12:00 AM UTC

Fedora 41 and Rawhide outage on public ranch - 0xe2 breaking ansible

Resolved after 95h 15m
November 9, 2024 at 12:00 AM UTC

Red Hat Ranch / BaseOS CI Outage - provisionig issues during the weekend

Resolved after 58h 0m
November 8, 2024 at 8:00 AM UTC

Misconfigured worker in public ranch

Resolved after 7h 0m
November 7, 2024 at 8:00 AM UTC

Testing Farm 2024-10.1 released

Resolved after 96h 0m
November 6, 2024 at 2:30 PM UTC

Testing Farm API down

Resolved after 45m
November 4, 2024 at 6:00 PM UTC

Gitlab CEE outage affecting jobs run on BaseOS CI and Red Hat ranch.

Resolved after 9h 0m
November 4, 2024 at 6:00 PM UTC

Fedora 41 outage on Red Hat ranch

Resolved after 18h 0m
November 1, 2024 at 12:00 AM UTC

Fedora 41 outage on public ranch.

Resolved after 15h 0m
November 1, 2024 at 12:00 AM UTC

BaseOS CI and Red Hat Ranch affected by provisioning outages. BaseOS CI down.

Resolved after 9h 0m
October 31, 2024 at 12:00 PM UTC

BaseOS CI and Red Hat Ranch affected by OpenShift outage

Resolved after 12h 0m
October 15, 2024 at 5:00 PM UTC

BaseOS CI suspended for changes in UMB certificate.

Resolved after 6h 0m
October 1, 2024 at 10:00 AM UTC

Public Ranch Outage - no archive disk space

Resolved after 60m
September 16, 2024 at 12:00 AM UTC

Public Ranch - broken Fedora repositories

Resolved after 19h 0m
September 3, 2024 at 8:00 PM UTC

Red Hat Ranch / BaseOS CI Outage - provisionig issues

Resolved after 17h 30m
August 30, 2024 at 8:30 AM UTC

Red Hat Ranch Outage - large logs consuming all the space. Testing-Farm dispatcher is disabled

Resolved after 5h 0m
August 28, 2024 at 10:29 AM UTC

Public ranch outage - workers not able to connect to Artemis.

Resolved after 3h 52m
August 20, 2024 at 6:00 PM UTC

Fedora 40 small disk size

Resolved after 206h 0m
August 13, 2024 at 3:40 PM UTC

Red Hat Ranch and BaseOS-CI Outage - dns misconfiguration, some crucial sites not accessible

Resolved after 9h 10m
August 8, 2024 at 4:00 AM UTC

Testing Farm API outage and longer queues on both ranches.

Resolved after 2h 15m
August 5, 2024 at 4:00 AM UTC

All Fedora images outdated, public ranch requests processing paused.

Resolved after 9h 0m
August 2, 2024 at 8:00 AM UTC

Fedora Rawhide outdated image

Resolved after 60m
July 26, 2024 at 12:00 AM UTC

Beaker PPC64 Outage - large queue because of limited hardware

Resolved after 168h 0m
July 25, 2024 at 1:00 PM UTC

Public and Redhat Ranch Outage - broken deployment

Resolved after 6h 45m
July 23, 2024 at 6:00 AM UTC

Public Ranch Outage - artifacts out of space

Resolved after 4h 0m
July 22, 2024 at 12:16 AM UTC

Public Ranch Outage - complete outage, request will stay queued or error out

Resolved after 9h 1m
July 16, 2024 at 12:00 AM UTC

Cyborg Loki is down

Resolved after 11h 30m
June 17, 2024 at 1:00 PM UTC

Public Ranch - Large queue

Resolved after 357h 0m
May 28, 2024 at 6:00 PM UTC

Red Hat Ranch and BaseOS-CI Outage - broken krb infra

Resolved after 16h 0m
May 27, 2024 at 12:00 AM UTC

Red Hat Ranch Outage - large queue, jobs getting delayed

Resolved after 12h 0m
May 19, 2024 at 6:00 AM UTC

Red Hat Ranch Outage - large queue, jobs getting delayed

Resolved after 14h 0m
May 15, 2024 at 8:00 PM UTC

Public Ranch Outage - possible large queue due to deadlocked workers

Resolved after 23h 30m
May 9, 2024 at 4:00 PM UTC

Public Ranch Outage - artifacts storage out of space

Resolved after 3h 30m
May 6, 2024 at 12:00 AM UTC

Public Ranch Outage - F40 images expired

Resolved after 7h 30m
April 30, 2024 at 6:00 PM UTC

Public Ranch Outage - large queue, high error rate due to broken workers

Resolved after 3h 30m
April 25, 2024 at 12:00 AM UTC

Public Ranch Outage - Random issues with access to koji

Resolved after 271h 15m
April 25, 2024 at 12:00 AM UTC

Public Ranch Outage - Broken Fedora Rawhide images

Resolved after 11h 0m
April 15, 2024 at 12:00 AM UTC

Public Ranch Outage - Expired images

Resolved after 8h 30m
March 29, 2024 at 12:00 AM UTC

Public Ranch Outage - Fedora 40 expired

Resolved after 17h 0m
March 28, 2024 at 12:00 AM UTC

Red Hat Ranch Degraded

Resolved after 22h 30m
March 28, 2024 at 12:00 AM UTC

Public Ranch Outage - Expired images

Resolved after 9h 30m
March 19, 2024 at 12:00 AM UTC

Public Ranch Outage - Artemis requests in error state

Resolved after 11h 0m
March 12, 2024 at 3:00 PM UTC

Long queues in Public and Red Hat ranch

Resolved after 29h 0m
March 8, 2024 at 8:00 AM UTC

Public Ranch artifact storage ran out of disk space

Resolved after 5h 0m
March 7, 2024 at 10:00 AM UTC

Public Ranch large queue - expect delays

Resolved after 6h 0m
March 5, 2024 at 3:00 PM UTC

Public Ranch large queue - expect delays

Resolved after 6h 0m
February 15, 2024 at 5:00 PM UTC

Public Ranch large queue - expect delays

Resolved in under a minute
February 2, 2024 at 8:00 AM UTC

BaseOS CI suspended for urgent jenkins controller upgrade.

Resolved after 60m
January 3, 2024 at 10:47 AM UTC

BaseOS CI suspended for urgent jenkins controller upgrade.

Resolved after 1h 43m

2023 (41)

December 10, 2023 at 12:00 AM UTC

Public Ranch Outage - Artemis requests in error state

Resolved after 35h 0m
December 2, 2023 at 12:00 AM UTC

Public Ranch Outage

Resolved after 3h 0m
November 29, 2023 at 12:00 AM UTC

Public Ranch reservations are broken

Resolved after 202h 0m
November 23, 2023 at 12:00 AM UTC

Fedora CI installability test affected by systemd/selinux issue.

Resolved after 1882h 0m
November 20, 2023 at 12:00 AM UTC

Testing Farm Public Ranch outage due to expired images.

Resolved after 10h 0m
November 19, 2023 at 6:00 AM UTC

Testing Farm Red Hat Ranch outage during the weekend

Resolved after 27h 0m
November 12, 2023 at 12:00 AM UTC

BaseOS CI limited due to outage due to BRQ network maintanance.

Resolved after 36h 0m
November 7, 2023 at 3:00 PM UTC

BaseOS CI and Testing Farm Red Hat Ranch outage due to Beaker unavailability

Resolved after 4h 0m
September 30, 2023 at 4:00 AM UTC

Expired images on Public ranch.

Resolved after 48h 0m
September 25, 2023 at 12:00 AM UTC

Testing Farm Red Hat ranch and BaseOS CI outage for RHEL9 and RHEL-8.1.0

Resolved after 16h 30m
September 21, 2023 at 12:00 AM UTC

BaseOS CI outage due to network issues

Resolved in under a minute
September 18, 2023 at 2:00 PM UTC

Console logs are disabled due to a discovered bug

Resolved in under a minute
September 17, 2023 at 8:00 AM UTC

Red Hat Ranch outage during weekend

Resolved after 8h 0m
September 15, 2023 at 8:00 AM UTC

Console log has wrong link in results viewer or not available at all.

Resolved after 274h 0m
September 11, 2023 at 8:00 AM UTC

Composes not updating on Red Hat ranch

Resolved in under a minute
September 6, 2023 at 8:00 AM UTC

Cyborg Loki is being migrated to S3 storage and better deployment

Resolved after 2788h 0m
August 21, 2023 at 8:00 AM UTC

Beaker testing blocked on Red Hat Ranch and BaseOS CI

Resolved after 4h 0m
August 7, 2023 at 12:00 AM UTC

Large queue times on Red Hat Ranch

Resolved after 8h 22m
July 27, 2023 at 9:00 AM UTC

Red Hat Ranch affected by large queue

Resolved after 3h 11m
July 19, 2023 at 8:00 AM UTC

Public ranch testing blocked

Resolved after 3h 0m
July 18, 2023 at 8:00 AM UTC

Fedora Rawhide testing blocked on dnf5 breakage

Resolved after 6h 0m
June 22, 2023 at 9:00 AM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 528h 0m
June 21, 2023 at 12:30 PM UTC

Fedora Rawhide testing blocked on dnf5

Resolved after 4h 30m
June 1, 2023 at 10:30 AM UTC

Red Hat Ranch affected by RH infra outage

Resolved after 21h 30m
May 29, 2023 at 2:00 AM UTC

Expired Fedora-Rawhide images in Public Ranch

Resolved after 5h 0m
May 5, 2023 at 2:00 AM UTC

Unexpected outage due to new release 2023-04.1

Resolved after 10h 0m
May 2, 2023 at 8:00 AM UTC

Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Resolved in under a minute
April 28, 2023 at 4:00 AM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m
April 18, 2023 at 2:00 PM UTC

Public Ranch - unexpected outage

Resolved after 6h 0m
April 14, 2023 at 3:00 AM UTC

Public Ranch - expired images

Resolved in under a minute
April 14, 2023 at 12:00 AM UTC

CentOS Stream 9 aarch64 broken in Public Ranch

Resolved after 92h 0m
April 6, 2023 at 10:00 AM UTC

Red Hat Ranch - deploying 2023-03.1 release - large queue expected

Resolved after 4h 0m
April 3, 2023 at 8:00 AM UTC

Red Hat Ranch - provisioning outage

Resolved in under a minute
March 29, 2023 at 8:00 AM UTC

Red Hat Ranch - larger queue

Resolved after 60m
March 28, 2023 at 8:00 PM UTC

Public ranch - Artemis resource ended in 'error' state

Resolved after 13h 0m
March 16, 2023 at 8:00 AM UTC

CentOS Stream 8 broken in Public Ranch

Resolved after 99h 0m
March 10, 2023 at 8:00 AM UTC

Fedora-Rawhide broken in Red Hat Ranch

Resolved after 148h 0m
February 10, 2023 at 9:00 AM UTC

Red Hat internal infrastructure outage

Resolved after 33h 0m
January 30, 2023 at 4:00 PM UTC

BaseOS CI - large queue, expect delays.

Resolved after 338h 0m
January 25, 2023 at 10:00 AM UTC

Public Ranch - large queue, expect delays.

Resolved after 7h 0m
January 24, 2023 at 3:00 PM UTC

Public Ranch outage - requests stuck in queued

Resolved after 19h 0m

2022 (19)

December 6, 2022 at 8:00 AM UTC

BaseOS CI Outage

Resolved after 9h 0m
November 30, 2022 at 9:00 AM UTC

Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Resolved after 7h 30m
November 29, 2022 at 8:00 AM UTC

Red Hat Ranch outage - artifacts storage empty

Resolved after 2h 30m
November 22, 2022 at 2:00 PM UTC

Public Ranch Outage - artifacts storage out of inodes

Resolved after 6h 0m
October 31, 2022 at 8:00 AM UTC

BaseOS CI Outage - UMB message sending

Resolved after 4h 0m
October 10, 2022 at 8:00 AM UTC

Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Resolved after 30h 20m
September 22, 2022 at 6:00 AM UTC

Public Ranch outage - Copr API v2 is deprecated

Resolved after 18h 30m
August 12, 2022 at 6:00 PM UTC

BaseOS CI Outage

Resolved after 67h 0m
August 11, 2022 at 8:00 AM UTC

CentOS Stream Koji Outage

Resolved after 2h 0m
August 1, 2022 at 11:00 AM UTC

Cyborg Development Cluster outage

Resolved after 2h 0m
June 16, 2022 at 9:00 AM UTC

CentOS Stream 9 repositories broken in CI

Resolved after 6h 0m
June 2, 2022 at 8:00 AM UTC

Cyborg Development Cluster credentials outage

Resolved after 7h 0m
March 25, 2022 at 12:00 PM UTC

Covscan testing blocked on BaseOS CI

Resolved after 72h 0m
February 8, 2022 at 12:00 PM UTC

rpminspect fails with "internal libpod error"

Resolved after 1441h 0m
February 5, 2022 at 8:00 PM UTC

Fedora CI outage - AWS images are gone

Resolved after 49h 25m
January 31, 2022 at 10:00 PM UTC

Broken rpmdeplint and rpminspect in Fedora CI

Resolved after 40h 30m
January 27, 2022 at 10:30 AM UTC

Large queue times on Red Hat Ranch

Resolved after 698h 22m
January 3, 2022 at 5:00 PM UTC

Large queue times on Public Ranch

Resolved after 27h 0m
January 3, 2022 at 12:17 AM UTC

Known issues  ℹ

There are currently no known issues, if you have troubles, contact us :) Recently resolved Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems …

2021 (3)

December 29, 2021 at 11:00 AM UTC

Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Resolved after 109h 12m
December 21, 2021 at 4:00 PM UTC

Issues with provisioning in Red Hat Ranch

Resolved after 20m
December 19, 2021 at 8:00 PM UTC

Issues Testing Fedora Rawhide

Resolved after 16h 0m

© Testing Farm Status, 2025   •   Back to top

We continuously monitor the status of our services and if there are any interruptions, a note will be posted here.

Powered by cState

⚡ Subscribe via RSS — to all updates