Red Hat Ranch on Testing Farm Status

[Resolved] Red Hat ranch requests are broken due to Artemis

Mon, 04 May 2026 12:30:00 +0000

Outage | 2026-05-04 12:30:00 UTC

Our provisioner went down. We are investigating why. RH ranch queue was suspended but a lot of jobs ended in error state.

Resolved | 2026-05-05 06:20:00 UTC

The outage is resolved. Seems everything is back to normal now.

[Resolved] Elevated API errors

Mon, 20 Apr 2026 15:19:00 +0000

Outage | 2026-04-20 15:19:00 UTC

We are seeing elevated API errors, we are investigating the issue.

Update | 2026-04-20 16:14:00 UTC

This should be better now.

Resolved | 2026-04-20 17:52:00 UTC

Resolving, the issue is gone

[Resolved] Testing Farm RH ranch not processing job from April 18th

Mon, 20 Apr 2026 06:38:00 +0000

Outage | 2026-04-20 06:38:00 UTC

Unfortunatley we forgot to rotate one of th tokens and the RH ranch is not processing any jobs from Saturday.

Update | 2026-04-20 08:31:00 UTC

This is now resolved. We canceld all hummingbird requests, the team will restart them. Others except java testing are restarted in place.

Resolved | 2026-04-20 15:18:00 UTC

This is resolved now

[Resolved] Errors when regenerating tokens via the UI

Thu, 16 Apr 2026 11:37:00 +0000

Outage | 2026-04-16 11:37:00 UTC

We are aware of an issue when regenerating tokens, we are looking into the issue.

Resolved | 2026-04-20 16:15:00 UTC

This issue should be resolved for some time now

[Resolved] Outdated image mode

Thu, 16 Apr 2026 09:38:00 +0000

Outage | 2026-04-16 09:38:00 UTC

Base images for image mode are delayed, leading to skipped update in testing farm.

Resolved | 2026-05-05 08:24:00 UTC

This should be resolved now.

[Resolved] Testing Farm container based testing is broken

Wed, 25 Mar 2026 15:00:00 +0000

Outage | 2026-03-25 15:00:00 UTC

We are sorry, the rotation of secrets broken our quay.io creds.

Resolved | 2026-03-26 08:57:00 UTC

Resolved. Testing Farm container testing is back.

[Resolved] All Beaker pools are disabled due to Beaker Scheduled Maintenance

Tue, 24 Mar 2026 13:09:00 +0000

Outage | 2026-03-24 13:09:00 UTC

All Beaker pools are disabled. Estimated time - 4 hours.

Resolved | 2026-03-24 16:00:00 UTC

The maintenance has ended

[Resolved] Testing Farm 2026-03.2 being deployed

Thu, 19 Mar 2026 19:18:00 +0000

Notice | 2026-03-19 19:18:00 UTC

We are deploying new Testing Farm version with tmt-1.69, stay tuned. Small downtime of API is expected, otherwise no other issues should arise. We will update this status entry if needed. ETA 4 hours.

Update | 2026-03-20 16:02:00 UTC

There was a small bug failing ~20 jobs in RH ranch causing Ansible to fail with: The conditional check ‘ansible_pkg_mgr in [‘unknown’, ‘dnf’]’ failed

Update | 2026-03-21 22:40:00 UTC

Testing Farm is now deployed, we will send out the release notes on Monday.

Update | 2026-03-21 22:42:00 UTC

There was an issue in Ansible setup for Fedora in Red Hat ranch, some of the jobs have failed, please restart them if you encountered Could not import the libdnf5 python module using /var/opt/standalone-python/bin/python3.

Resolved | 2026-03-23 19:02:00 UTC

Testing Farm was deployed! Enjoy!

[Resolved] ibmcloud-ppc64le cloud disabled

Mon, 16 Mar 2026 11:00:00 +0000

Outage | 2026-03-16 11:00:00 UTC

Issues with error instances that can’t be removed and new instance provisioning, investigation in progress. Pool is disabled.

Resolved | 2026-03-27 11:00:00 UTC

Error instances cleaned up, pool re-reenabled with 30 instances limit.

[Resolved] Beaker's disrupting Red Hat ranch.

Thu, 12 Mar 2026 16:07:00 +0000

Outage | 2026-03-12 16:07:00 UTC

The Beaker scheduler have issues which affects Red Hat ranch’s Beaker pool. Expect delays in guest provisioning.

Resolved | 2026-03-12 19:30:00 UTC

Seems the Beaker scheduler is back and running. The incident is resolved.

[Resolved] Testing Farm outage - Cannot execute the rule

Thu, 05 Mar 2026 21:00:00 +0000

Outage | 2026-03-05 21:00:00 UTC

Both ranches are suffering outage after the most recent release deployment. A mistake in configuration is causing pipelines to fail. We are currently investigating why the corresponding fix have not been deployed properly.

Update | 2026-03-06 10:33:00 UTC

It was affecting Public Ranch too. Please restart your failed requests. The issue was fixed by reverting change. The issue was not affecting requests testing against containers.

Resolved | 2026-03-06 10:00:00 UTC

Hotfix deployed and new request are working.

[Resolved] IBM Cloud ppc64le outage

Mon, 02 Mar 2026 09:06:00 +0000

Outage | 2026-03-02 09:06:00 UTC

IBM cloud ppc64le provisioning is broken. All the ppc64le workloads from Testing Farm will land in Beaker. If you can postpone or limit your ppc64le testing via Testing Farm and BaseOS CI for testing unrelated to RC baseline testing, please do so.

Update | 2026-03-03 17:11:00 UTC

So the ppc64le queue is not exploding, we were able to unblock some ppc64le machines and seems IBM cloud is getting back. More update in few hours.

Update | 2026-03-05 11:01:00 UTC

IBM Cloud ppc64le is enabled back. Beaker ppc64le queue times are back to normal.

Update | 2026-03-05 12:09:00 UTC

Resolving.

Resolved | 2026-03-05 12:09:00 UTC

Resolved

[Resolved] RHEL-9.9.0 and RHEL-10.3 image mode testing blocked

Tue, 24 Feb 2026 14:01:00 +0000

Outage | 2026-02-24 14:01:00 UTC

The image mode testing of RHEL-9.9.0 and RHEL-10.3 is blocked on unavailability of image mode rhel container images. See https://issues.redhat.com/browse/BIFROST-1113 for details.

Resolved | 2026-03-12 16:43:00 UTC

The images are updated. The outage is closed.

[Resolved] Testing Farm 2026-02.2 was deployed on 2026-02-19

Thu, 19 Feb 2026 18:58:00 +0000

Outage | 2026-02-19 18:58:00 UTC

New Testing Farm version was deployed, but we will send out release notes on Monday, as we did not make it. For a previous of release notes see: https://testing-farm.gitlab.io/-/docs/root/-/jobs/13185451238/artifacts/preview/Testing%20Farm/0.1/releases/2026-02.2.html

Resolved | 2026-02-23 10:21:00 UTC

Resolving, new release is deployed fully.

[Resolved] RHEL-9.9, RHEL-10.1, RHEL-10.2 and RHEL-10.3 compose update blocked

Thu, 19 Feb 2026 10:22:00 +0000

Outage | 2026-02-19 10:22:00 UTC

RHEL-9.9.0, RHEL-10.1, RHEL-10.2 and RHEL-10.3 are blocked due to multiple issues. See https://issues.redhat.com/browse/TFT-4367 for details. No ETA yet, but the issues are escalated as blockers.

Resolved | 2026-02-23 23:32:00 UTC

Resolving, package mode images are now available.

[Resolved] Broken Fedora repositories

Tue, 10 Feb 2026 23:00:00 +0000

Outage | 2026-02-11 00:00:00 UTC

We are experiencing outage of repositories on Fedora images.

Issue is tracked in: https://forge.fedoraproject.org/infra/tickets/issues/13138

Updating and loading repositories:
 Fedora 43 openh264 (From Cisco) - x86_ 100% |   8.6 KiB/s |  10.8 KiB |  00m01s
Failed to download metadata
 Error loading local metadata for repository "updates"
  Librepo error: Checksum error /var/cache/libdnf5/updates-431245df5652aba0/repodata/72324174538178d41d9e8bc090d9eb64666f071a34c516937f13e6932546a129-filelists.xml.zck: Unable to read zchunk lead

Resolved | 2026-02-17 01:02:00 UTC

Resolving, seems the issue was addressed and we do not see it anymore.

[Resolved] RHEL-8.10-ppc64le outage due to standalone python

Tue, 10 Feb 2026 00:00:00 +0000

Outage | 2026-02-10 00:00:00 UTC

We are experiencing outage of RHEL-8.10-ppc64le due to standalone python runtime.

Issue is tracked in: https://issues.redhat.com/browse/TFT-4325

Resolved | 2026-03-23 21:15:00 UTC

Resolved in TF 2026-03.2 release

[Resolved] Fedora 44 not available in Testing Farm

Mon, 09 Feb 2026 11:46:00 +0000

Outage | 2026-02-09 11:46:00 UTC

The import is in progress after branching. We are working it out. ETA later today.

Resolved | 2026-02-10 02:45:00 UTC

Fedora 44 is now available.

[Resolved] Testing Farm 2026-02.1 deployment in progress

Mon, 02 Feb 2026 10:46:00 +0000

Outage | 2026-02-02 10:46:00 UTC

We are deploying a new TF release, brief outages of API might happen during this time.

Update | 2026-02-02 18:51:00 UTC

Deployment is half done, and ongoing to the next day

Resolved | 2026-02-04 15:59:00 UTC

Testing Farm 2026-02.1 is fully deployed. Resolving.

[Resolved] Image Mode doesn't work correctly on the redhat ranch

Thu, 29 Jan 2026 10:21:00 +0000

Outage | 2026-01-29 10:21:00 UTC

We discovered problems with image-mode composes mapping. Testing Farm may use non-image-mode images for image mode testing. We are working on a fix.

Update | 2026-02-02 18:38:00 UTC

The outage is still not resolved, we are looking into the issue.

Resolved | 2026-02-02 11:47:00 UTC

Resolved.

[Resolved] Red Hat Ranch and BaseOS CI not working due to provisioner outage

Wed, 28 Jan 2026 00:00:00 +0000

Outage | 2026-01-28 00:00:00 UTC

Provisioner does not work, blocking processing all requests on Red Hat Ranch and BaseOS CI.

Update | 2026-01-28 09:00:00 UTC

The outage was resolved, the queue is going down.

Resolved | 2026-01-28 12:00:00 UTC

The queue is now gone.

[Resolved] IBM Cloud outage

Wed, 21 Jan 2026 00:00:00 +0000

Outage | 2026-01-21 00:00:00 UTC

Due to various problems with IBM Cloud, we have had to disable it’s usage temporarily until these problems are resolved. Machines from Beaker will be used and you will notice a significant delay completing request for the architectures: s390x, ppc64le

Resolved | 2026-01-29 17:55:00 UTC

Outage was resolved. Pool enabled back.

[Resolved] Compose update blocked due to issues in IBM cloud

Tue, 20 Jan 2026 09:00:00 +0000

Outage | 2026-01-20 09:00:00 UTC

We are having issues pushing through update on IBM cloud, this is blocking all compose updates. IBM support is looking into the issue.

Resolved | 2026-02-02 10:45:00 UTC

This is now resolved.

[Resolved] Elevated API errors in Testing Farm

Mon, 19 Jan 2026 09:00:00 +0000

Outage | 2026-01-19 09:00:00 UTC

We are experiecing read timeouts on our API. Issue is under investigation.

Update | 2026-01-19 20:00:00 UTC

We are patching API to better recycle stale connections

Update | 2026-01-19 20:21:00 UTC

Issue should be addressed, we are monitoring the situation. https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 20:34:00 UTC

Things are back to stable, we hotpatched production with: https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 21:11:00 UTC

The issues is back :( we are looking further

Resolved | 2026-01-19 23:33:00 UTC

Resolving, we did nto see this anymore after applying the fixes.

[Resolved] Redoc documentation shows empty page

Thu, 08 Jan 2026 14:03:00 +0000

Outage | 2026-01-08 14:03:00 UTC

We have a dependencies problem which affects https://api.testing-farm.io/redoc page rendering it empty. Please use https://api.testing-farm.io/docs instead for now. We are working on a fix

Resolved | 2026-01-15 19:03:00 UTC

resolved with 2026-01.1 release

[Resolved] Provisioning errors due to missing s390x images

Fri, 12 Dec 2025 10:26:00 +0000

Outage | 2025-12-12 10:26:00 UTC

Our compose-update let through some composes with missing s390 images in IBM cloud, breaking provisioning. We are looking into the situation, for more information see: https://redhat-internal.slack.com/archives/C03BRN71JAF/p1765792914415559

Resolved | 2025-12-15 12:25:00 UTC

Errors resolved and images synced.

[Resolved] Beaker outage affecting provisioning

Tue, 09 Dec 2025 07:59:00 +0000

Outage | 2025-12-09 07:59:00 UTC

We are seeing elevated error rate when provisioning Beaker, investigation is in progress.

Update | 2025-12-09 08:40:00 UTC

Seems things look more stable now, reported to #help-beaker channel for investigation of the 500s we and users directly interacting with Beaker API were seeing: https://redhat-internal.slack.com/archives/C04KRPW1HJ4/p1765266395518069

Update | 2025-12-09 18:26:00 UTC

Resolved, it was caused by Beaker going out of disk space :(

Resolved | 2025-12-09 20:06:00 UTC

Resolving …

[Resolved] Fedora 42 is unavailable in Beaker

Wed, 03 Dec 2025 08:33:00 +0000

Outage | 2025-12-03 08:33:00 UTC

Fedora-42 and Fedora-42-Updated are unavailable in Beaker due to problems booting the system. Investigation is on-going. Thank you for your patience.

Update | 2025-12-03 15:32:00 UTC

Fedora-43 also affected.

Update | 2025-12-10 18:27:00 UTC

Seems the issue is caused by an SELinux problem, we are gathering more info, cannot yet confirm the issue is resolved.

Update | 2025-12-19 03:30:00 UTC

The issue still persists, for more information see: https://redhathub.service-now.com/hub?id=standard_ticket&table=universal_request&sys_id=ed719a372fe13e903538cd3bcfa4e319

Resolved | 2026-01-15 19:02:00 UTC

Resolved on Beaker side, see: https://redhat-internal.slack.com/archives/C04KRPW1HJ4/p1768392931745919

[Resolved] Testing Farm updating AWS DNS setup to support DNSSEC, failover to Openstack and Beaker

Thu, 06 Nov 2025 21:00:00 +0000

Outage | 2025-11-06 21:00:00 UTC

For more information see: https://groups.google.com/a/redhat.com/g/rhel-devel/c/-46R4axLj_s

Resolved | 2025-11-06 23:45:00 UTC

The setup is complete, unfortunately the changes did not have effect we hoped for. See https://redhat-internal.slack.com/archives/C03BRN71JAF/p1762277234611719 Slack for more information. Resolving the outage.

[Resolved] Beaker pools not working

Fri, 31 Oct 2025 09:05:00 +0000

Outage | 2025-10-31 09:05:00 UTC

Due to a breaking change in Artemis v0.0.81 the Beaker pools became broken and erroring out all the time.

Update | 2025-11-03 09:07:00 UTC

The problem was identified early Monday, and the fix is getting deployed to mitigate this problem. Next to it we are extending our monitoring to cover the problem.

Update | 2025-11-03 10:44:00 UTC

The issue is resolved, we are monitoring the problem.

Update | 2025-11-04 03:15:00 UTC

Issue should be resolved

Resolved | 2025-11-06 20:52:00 UTC

Resolving, things are stable now.

[Resolved] AWS world-wide outage affecting Testing Farm

Mon, 20 Oct 2025 09:29:00 +0000

Outage | 2025-10-20 09:29:00 UTC

The world wide AWS outage is affecting Testing Farm. We are looking how to mitigate the issue, but be prepared for delays in testing. Public Ranch is unaffected as it is hosted in us-east-2.

Update | 2025-10-20 18:00:00 UTC

Issues are persisting, things looked better, now it is worse and we had to disable AWS alltogether. Expect delays.

Resolved | 2025-10-21 01:04:00 UTC

Outage is resolved. AWS should be back to normal operations. We have enabled back the AWS pools in general routing.

[Resolved] quay.io is down

Mon, 20 Oct 2025 07:51:00 +0000

Outage | 2025-10-20 07:51:00 UTC

We are aware of quay.io outage, we are gatheting more information. This will affect our services, but the real extent is not yet known.

Update | 2025-10-20 07:58:00 UTC

Seems connected to the AWS ouatage in us-east-1: https://health.aws.amazon.com/health/status

Update | 2025-10-20 09:10:00 UTC

Outage was resolved by routing traffic to us-east-2.

Resolved | 2025-10-20 09:10:00 UTC

Resolving

[Resolved] Testing Farm 2025-09.1 was deployed

Fri, 17 Oct 2025 11:23:00 +0000

Outage | 2025-10-17 11:23:00 UTC

We are preparing release notes.

Update | 2025-11-12 12:41:00 UTC

Release notes are available:

https://docs.testing-farm.io/Testing%20Farm/0.1/releases/2025-09.1.html

Resolved | 2025-11-12 12:43:00 UTC

Resolving

[Resolved] Brew outage

Fri, 17 Oct 2025 09:00:00 +0000

Outage | 2025-10-17 09:00:00 UTC

Maintenance affecting brew, dist-git, MBS

Update | 2025-10-20 07:19:00 UTC

Brew is working again, resolving.

Resolved | 2025-10-20 09:11:00 UTC

Resolving

[Resolved] Fedora and CentOS stream compose broken due to download.bos down

Thu, 02 Oct 2025 10:40:00 +0000

Outage | 2025-10-02 10:40:00 UTC

We are aware of an issue affecting Fedora and CentOS Stream testing in Red Hat ranch. This is due to download.bos outage, we are working on a workaround.

Resolved | 2025-10-08 23:22:00 UTC

This is resolved, problem was addressed

[Resolved] Default instance for s390x has 10GiB disk size

Fri, 19 Sep 2025 09:31:00 +0000

Outage | 2025-09-19 09:31:00 UTC

Due to a misconfiguration the default disk size is 10GiB for s390x machines which do not specify the disk hardware requirement. To workaround, please specify the disk hardware requirement. In this case your request will be routed to Beaker instead of the default IBM cloud. We are working hard to resolve this issue, but it might take some significant time as we need to adjust our Openshift deployment due to hitting maximum configmap size limit.

Update | 2025-09-22 10:46:00 UTC

The issue was resolved, the disk size should be now 120GiB by default.

Resolved | 2025-09-22 10:47:00 UTC

Resolved, per above comment.

[Resolved] Beaker provisioning issues

Mon, 01 Sep 2025 11:50:00 +0000

Outage | 2025-09-01 11:50:00 UTC

There is a wide problem for Beaker causing provisioning issues. Affecting at least s390x. See https://redhat.service-now.com/surl.do?n=INC3762615 for details.

Resolved | 2025-09-03 17:32:00 UTC

Outage was resolved in the evening of 3rd September

[Resolved] Red Hat Gitlab Access Limited

Wed, 13 Aug 2025 09:00:00 +0000

Outage | 2025-08-13 09:00:00 UTC

The Gitlab team is workning to resolve this. You may see HTTP 429 responses in your logs and timeouts. If urgent, please restart the test job.

Resolved | 2025-08-14 07:03:00 UTC

Seems the outage is resolved. Gitlab.cee and pipelines work now well.

[Resolved] Compose update delayed due to gitlab workers remaining in pending state.

Wed, 23 Jul 2025 09:15:00 +0000

Outage | 2025-07-23 09:15:00 UTC

Intermitent issue. May have minimal impact, but will affect users requiring today’s composes.

Resolved | 2025-07-23 17:26:00 UTC

gitlab workers are running jobs again now with no action from the Testing Farm team.

[Resolved] BaseOS CI and Testing Farm testing blocked on artifacts.osci.redhat.com outage

Wed, 16 Jul 2025 22:00:00 +0000

Outage | 2025-07-16 22:00:00 UTC

The artifacts.osci.redhat.com was migrated to ITUP infrastructure, and the NFS shares did not make it. Basically archiving is impossible, and that is making all the jobs blocked. No ETA currently, folks will be looking into it in the morning.

Update | 2025-07-17 09:25:00 UTC

We are working with ITUP to revive the artifacts storage. No ETA available yet.

Update | 2025-07-17 14:35:00 UTC

We are working with OSCI and ITUP on creating a new NFS volume and migrating a part of artifacts data. No ETA available yet.

Update | 2025-07-17 19:00:00 UTC

The investigation and work with OSCI and ITUP continues. Unfortunately, no ETA on resolving the issue yet :(

Update | 2025-07-18 08:00:00 UTC

The artifact storage has been migrated once again, and the NFS response is back to where it needs to be. Both Red Hat Ranch and Base OS CI were unblocked and started crunching their queue. We are monitoring the situation before the outage gets resolved properly.

Resolved | 2025-07-18 10:30:00 UTC

Resolved. The storage is stable, and the queue is getting smaller.

[Resolved] OpenStack outage

Mon, 14 Jul 2025 08:00:00 +0000

Outage | 2025-07-14 08:00:00 UTC

OpenStack is not operational and affects testing on machines managed by OpenStack.

Resolved | 2025-07-15 21:00:00 UTC

Image listing problem found and resolved.

[Resolved] src.fedoraproject.org is down

Thu, 26 Jun 2025 10:00:00 +0000

Outage | 2025-06-26 10:00:00 UTC

DC reloctation outage expected. Pleases see status here https://status.fedoraproject.org/ CentOS pkg repos are also affected.

Resolved | 2025-07-14 07:53:00 UTC

The outage is resolved. All Fedora systems are back to normal. Resolving outage.

[Resolved] Stuck artemis in redhat ranch

Sat, 14 Jun 2025 12:09:00 +0000

Outage | 2025-06-14 12:09:00 UTC

provisioning is stuck and queue is growing

Resolved | 2025-06-14 13:21:00 UTC

unblocked by redeployment, queue should be processed soon

[Resolved] Red Hat Ranch and BaseOS CI not working due to provisioner outage

Fri, 13 Jun 2025 22:00:00 +0000

Outage | 2025-06-13 22:00:00 UTC

Provisioner does nto work, blocking processing all requests on Red Hat Ranch and BaseOS CI

Update | 2025-06-15 11:29:00 UTC

The root cause of the issue was identified and hotfixed. We have a fix so we mitigate this issue to happen again. If you encounter any unexpected provisioning or timeout errors in your test requests, please restart them via your CI system or manually.

Resolved | 2025-06-17 08:23:00 UTC

The outage was resolved on 15th, closing outage.

[Resolved] IT resilience testing multiple SP services down

Fri, 13 Jun 2025 13:00:00 +0000

Outage | 2025-06-13 13:00:00 UTC

tps-server

Resolved | 2025-06-13 21:00:00 UTC

Testing complete

[Resolved] Testing Farm testing disabled on s390x and ppc64le

Wed, 11 Jun 2025 10:50:00 +0000

Outage | 2025-06-11 10:50:00 UTC

Due to the state of Beaker, see https://redhat.service-now.com/surl.do?n=INC3687996, we have disabled testing on s390x and ppc64le. All requests for these architectures won’t be accepted on the API level.

Resolved | 2025-06-13 09:25:00 UTC

Issues with beaker should be resolved, s390x is not aborting any more.

[Resolved] ppc64le testing blocked

Mon, 09 Jun 2025 07:57:00 +0000

Outage | 2025-06-09 07:57:00 UTC

becaues of long queue time in beaker, ppc64le is blocked

Update | 2025-06-09 09:43:00 UTC

many ppc64le requests running for long time canceled

Resolved | 2025-06-13 08:14:00 UTC

queue is almost empty, ppc64le testing shoud work well. In case of aborted job because of timeout, restart your tests

[Resolved] workflow-tomorrow tests will not run

Wed, 04 Jun 2025 08:00:00 +0000

Outage | 2025-06-04 08:00:00 UTC

Due to decommision of REDHAT.COM kerberos realm, workflow-tomorrow tests do not run.

Update | 2025-06-05 17:49:00 UTC

new keytab is deployed and tested, restarting affected jobs. There is ~ 300 jobs, so it will take many hours to process

Update | 2025-06-05 20:11:00 UTC

wow works again, most of affected jobs restarted, pending issue with ODCS access

Resolved | 2025-06-06 13:00:00 UTC

new keytab is working as expected

[Resolved] Beaker outage

Wed, 04 Jun 2025 07:33:00 +0000

Outage | 2025-06-04 07:33:00 UTC

Beaker machines are unavailable due to a connectivity problem. Investigation is ongoing.

Update | 2025-06-04 08:50:00 UTC

Authentication update needed.

Update | 2025-06-04 08:56:00 UTC

Authentication updated… testing.

Update | 2025-06-04 11:35:00 UTC

Confirmed working.

Resolved | 2025-06-04 11:25:00 UTC

Change deployed to production.

[Resolved] RHEL-9.6.0 and RHEL-10.0 image update images are outdated

Tue, 03 Jun 2025 10:00:00 +0000

Outage | 2025-06-03 10:00:00 UTC

The source images from bootc our outdated. Please see https://issues.redhat.com/browse/BIFROST-768 for details. We are blocked until this issue is resolved.

Resolved | 2025-06-30 09:07:00 UTC

The root case was fixed, images are not current. Resolving.

[Resolved] RHEL9 and RHEL10 compose update blocked on IBM cloud outage

Mon, 02 Jun 2025 13:00:00 +0000

Outage | 2025-06-02 13:00:00 UTC

IBM Cloud outage is blocking updating composes for RHEL9 and RHEL10. We are waiting for https://groups.google.com/a/redhat.com/g/outage-list/c/u93AOmuaKqM to be resolved to resume the process.

Update | 2025-06-04 10:20:00 UTC

The issue was resolved yesterday, but it is back again today :(

Resolved | 2025-06-11 08:20:00 UTC

Seems the issue is resolved.

[Resolved] Artemis issue with policies

Thu, 22 May 2025 09:07:00 +0000

Outage | 2025-05-22 09:07:00 UTC

Investigating root cause and reverting recent changes. Requests are stuck in routing and will be processed after revert

Update | 2025-05-22 11:52:00 UTC

Issue fixed, queue of guests in routing is going down

Resolved | 2025-05-22 12:57:00 UTC

All requests stuck in routing are now processed. In case of timeout, please restart your test.

[Resolved] Some rpminspect jobs terminated on out-of-memory

Wed, 21 May 2025 06:00:00 +0000

Outage | 2025-05-21 06:00:00 UTC

We are aware of an issue causing rpminspect jobs to be terminted on out-of-memory events even if they have abidiff check disabled. We have rolled back the oom monitoring and are working on a fix in https://issues.redhat.com/browse/TFT-3380. Please restart any jobs if you encounter any falsly terminated jobs. They should be passing now.

Resolved | 2025-05-26 18:24:00 UTC

The issue was addressed and only certain requests should be now termindated and the documentation should be correct. Resolving.

[Resolved] ansible-core breaks ansible-lint

Tue, 20 May 2025 07:42:00 +0000

Outage | 2025-05-20 07:42:00 UTC

Tooling affected by https://github.com/ansible/ansible-lint/issues/4611

Update | 2025-08-07 12:00:00 UTC

This has been mostly mitigated by forcing the ansible version to something before 2.19.

Update | 2025-08-18 09:12:00 UTC

Seems resolved for some time now.

Resolved | 2025-08-19 17:20:00 UTC

Resolving

[Resolved] Instances gone during test execution

Tue, 20 May 2025 06:00:00 +0000

Outage | 2025-05-20 06:00:00 UTC

The spot instances are getting lately terminated more then expected. We are moving to non-spot instances by default, see https://issues.redhat.com/browse/TFT-3371 for details.

Resolved | 2025-05-22 18:01:00 UTC

The issue should be resolved, we are now using non-spot instances by default.

[Resolved] Gitlab.com outage causing various requests to fail

Mon, 19 May 2025 21:00:00 +0000

Outage | 2025-05-19 21:00:00 UTC

Gitlab.com has a wide outage, see https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/682bb3b0158484053205f59b. If the requests use gitlab.com, they might randomly fail on 500 and similar.

Resolved | 2025-05-20 18:00:00 UTC

Things look more stable now. Resolving.

[Resolved] Testing Farm Red Hat ranch high queue

Thu, 15 May 2025 15:41:00 +0000

Outage | 2025-05-15 15:41:00 UTC

We are experiencing high load on Testing Farm, expect delays on your requests.

Resolved | 2025-05-18 23:48:00 UTC

The queue should be back to normal now.

[Resolved] Beaker outage

Tue, 13 May 2025 14:18:00 +0000

Outage | 2025-05-13 14:18:00 UTC

Beaker outage affects all SUT/DUTs from Beaker.

Resolved | 2025-05-14 07:02:00 UTC

Beaker reboots complete.

[Resolved] BRQ beaker lab aborting jobs

Mon, 05 May 2025 11:00:00 +0000

Outage | 2025-05-05 11:00:00 UTC

jobs ara aborting because of disk space issue

Resolved | 2025-05-05 19:33:00 UTC

resolved by beaker team

[Resolved] Red Hat Ranch blocked on high queue

Wed, 23 Apr 2025 22:00:00 +0000

Outage | 2025-04-23 22:00:00 UTC

be patient until we resolve the issue and unblock the queue processing.

Update | 2025-04-24 12:37:00 UTC

The queue is getting better, see http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?orgId=1&refresh=1m&viewPanel=36 for details.

Resolved | 2025-04-25 21:04:00 UTC

Resolving the outage. Testing Farm is stable now. Note that there is still possibility of higher load, as we have users submitting hundreds of jobs at once.

[Resolved] RHEL-9.6.0 not schedulable on Testing Farm

Thu, 17 Apr 2025 15:24:00 +0000

Outage | 2025-04-17 15:24:00 UTC

deployment change. Please start your testing again if you were affected.

Update | 2025-04-22 11:28:00 UTC

Actually the problem affected only Testing Farm.

Update | 2025-04-22 11:30:00 UTC

We asked OSCI to rerun everything from 17th. So RHEL gating should get automatically retried.

Resolved | 2025-04-17 18:00:00 UTC

The compose was unblocked in Testing Farm. Outage resolved.

[Resolved] Some requests cancelled by an error.

Mon, 14 Apr 2025 09:52:00 +0000

Outage | 2025-04-14 09:52:00 UTC

Some of the requests might have canceled by admin error. Please restart your testing if your request was canceled unexpected. We are sorry for the inconvenience.

Resolved | 2025-04-14 10:00:00 UTC

Resolving this notice after 3 days.

[Resolved] Artifacts storage outage for BaseOS CI and Testing Farm

Fri, 11 Apr 2025 13:00:00 +0000

Outage | 2025-04-11 13:00:00 UTC

The artifacts storage was restarted and lost all mount points including Testing Farm and BaseOS CI artifacts storage.

Resolved | 2025-04-11 15:10:00 UTC

The used DNS servers were removed by IT last year. The servers were corrected, mounts points remounted and the outage should be cleared. We are planning to move away from this artifacts storage for better experience.

[Resolved] High load in testing-farm

Wed, 09 Apr 2025 16:00:00 +0000

Outage | 2025-04-09 16:00:00 UTC

High load and queue, expect delays.

Resolved | 2025-04-09 19:26:00 UTC

TestingFarm queue is processed.

[Resolved] Beaker Outage on Red Hat ranch

Tue, 08 Apr 2025 10:00:00 +0000

Outage | 2025-04-08 10:00:00 UTC

Red Hat ranch disruptance due to Beaker pool monthly reboot. ETA - 4 hours.

Update | 2025-04-08 21:57:00 UTC

We are investigating high queue most probably caused by the Beaker outage.

Update | 2025-04-08 23:25:00 UTC

In case you hit some weird errors, make sure to restart.

Update | 2025-04-09 16:26:00 UTC

Queue length is still over 4 hours for ppc64le

Update | 2025-04-10 02:31:00 UTC

Queue length is still over 6 hours for ppc64le. But all jobs on Testing Farm are processed and waiting for free machines in case they landed in Beaker. We are keeping open the outage for awareness.

Resolved | 2025-04-14 09:50:00 UTC

If you see some unexpected results, please restart your testing.

[Resolved] Large queue on Red Hat ranch

Mon, 07 Apr 2025 16:02:00 +0000

Outage | 2025-04-07 16:02:00 UTC

Expect larger queue, we are working on scaling up Testing Farm to deal with the queue.

Resolved | 2025-04-07 20:40:00 UTC

The queue is gone, pipeline should be back to normal.

[Resolved] RHEL-10.0 is expired on Red Hat Ranch

Mon, 07 Apr 2025 07:58:00 +0000

Outage | 2025-04-07 07:58:00 UTC

We are working to resolve the issue.

Update | 2025-04-07 09:42:00 UTC

Images are built, we are working on releasing them. ETA 1h.

Update | 2025-04-07 11:11:00 UTC

Updated compose is deployed to production.

[Resolved] Testing Farm outage due to deployment

Thu, 03 Apr 2025 07:30:00 +0000

Resolved 2025-04-03 9:00 UTC - Testing Farm 2025-01.4 is released. Queue getting crunched

Outage 2025-04-03 7:30 UTC - Both ranches are disabled due to messed up deployment. Expect long queue times.

[Resolved] Red Hat Ranch - expect delays for RHEL-9.5.0 and RHEL-10.0

Thu, 03 Apr 2025 04:00:00 +0000

Notice | 2025-04-03 04:00:00 UTC

Note that due to accidentally removed images for RHEL-10.0 and RHEL-9.5 you can get some unexpected delays. The issue should be addressed and we believe the jobs will start slowly resuming.

Resolved | 2025-04-03 16:00:00 UTC

Images were updated, resolving notice.

[Resolved] Planned brew outage

Fri, 28 Mar 2025 10:00:00 +0000

Outage 2025-03-28 10:00 UTC - Start of planned Brew outage window. BaseOS-CI is suspended, Red Hat Ranch is affected for manually triggered brew task testing

Update 2025-03-28 14:15 UTC - Some complication, unplanned outage right after end of planed one

Resolved 2025-03-29 12:00 UTC - Resuming queue (meanwhile jenkins controller has been upgraded)

[Resolved] Red Hat Ranch Outage - provisioning constrained

Mon, 17 Mar 2025 09:00:00 +0000

Resolved 2025-03-18 08:00 UTC - The problem was addressed, make sure to restart your testing if you were affected by the outage. Sorry for the incovenience.

Outage 2025-03-17 09:00 UTC - Red Hat Ranch testing is disrupted due to provisioning being affected by inablity to talk to services protected by Kerberos. This severly reduces the number of workers taking care of provisioning, and disables Beaker-backed provisioning.

[Resolved] Fedora broken on aarch64, s390x and ppc64le in Beaker

Mon, 10 Mar 2025 06:00:00 +0000

Outage | 2025-03-10 06:00:00 UTC

Fedora testing on s390x, ppc64le and aarch64 is broken. Please share your concern in https://redhat.service-now.com/surl.do?n=INC3438634.

Update | 2025-05-27 08:20:00 UTC

The issue is still present. We have users who are failing to conduct their testing in Fedora-Rawhide and Fedora-42.

Update | 2025-11-03 10:43:00 UTC

This issue is still present. Please update the service now help ticket.

Resolved | 2025-11-27 15:48:00 UTC

The outage is resolved finally. The distros should be installable without issues in Beaker.

[Resolved] Red Hat Ranch Outage - artifacts storage not accessible

Fri, 07 Mar 2025 10:00:00 +0000

Resolved 2025-03-07 16:00 UTC - The DNS records on artifacts.osci.redhat.com were outdated for months. Todays maintanance exposed this problem by causing an outage. Requests will not resume, please restart your requests.

Outage 2025-03-07 10:00 UTC - Red Hat Ranch testing is disrupted due to artifacts storage (artifacts.osci.redhat.com) inacessible. This causes the requests to end up with empty or incomplete artifacts results.

[Resolved] OpenStack outage in Red Hat ranch

Thu, 27 Feb 2025 12:00:00 +0000

Resolved 20205-02-27 18:00 UTC - Everything is back to normal.

Outage 2025-02-27 12:00 UTC - Red Hat ranch is disrupted due to OpenStack outage. Pool is not reachable and artifact pages are not accesible.

[Resolved] RHIVOS HW pool outage - several issues

Mon, 24 Feb 2025 08:28:04 +0000

Outage 2025-02-24 08:00 UTC - ride4-atc and rcar-s4-atc pools are discovered to be problematic.

Update 2025-02-24 11:00 UTC - rcar-s4-atc discovered to have been restarted automatically by systemd on Friday and did restart correctly. Fix is being worked on. ride4-atc pool investigation continues.

Update 2025-02-24 13:00 UTC - ride4-atc and rcar-s4-atc restored. ride4-atc pool also suffered catastrophic restarts. Investigation continues.

[Resolved] Beaker Outage - several issues

Wed, 12 Feb 2025 08:00:00 +0000

Outage 2025-02-12 08:00 UTC - New release of restraint broke reservation

Update 2025-02-17 20:00 UTC - Issue with restraint solved, but beaker is not in a good shape: long queues, many machines marked as broken

Update 2025-02-18 16:00 UTC - Beaker is getting well, but we have some composes outdated, new ones bloced by issue with installing qa-tools

Resolved 2025-02-19 09:00 UTC - All expired composes updated, everything works as expected, beaker queues < 4h

[Resolved] Beaker Outage

Tue, 11 Feb 2025 14:00:00 +0000

Outage 2025-02-11 14:00 UTC - All workloads run on Beaker are affected. The exact time the outage begin is not yet known.

Resolved 2025-02-11 18:00 UTC - Beaker updates/changes/restarts are finally complete.

[Resolved] Red Hat Ranch Outage - issue with Artemis deployment

Fri, 31 Jan 2025 18:00:00 +0000

Resolved 2025-02-03 02:00 UTC - All running jobs timed out, please restart your testing. Thinks should be stable now. We are looking into how to prevent the situation.

Outage 2025-01-31 18:00 UTC - Requests in Red Hat Ranch are stuck due to Artemis outage.

[Resolved] Red Hat Ranch Outage - issue with Artemis deployment

Fri, 17 Jan 2025 09:00:00 +0000

Outage 2025-01-17 09:00 UTC - Requests in Red Hat Ranch are stuck due to Artemis outage.

[Resolved] Red Hat Ranch Outage - issue with Artemis deployment

Thu, 16 Jan 2025 00:20:00 +0000

Resolved 2025-01-16 04:20 UTC - Artemis deployment recovered and working properly again

Outage 2025-01-16 00:20 UTC - Red Hat ranch blocked by artemis deployment issue ind OpenShift

[Resolved] Brew outage https://brewweb.engineering.redhat.com/brew/

Tue, 10 Dec 2024 09:00:00 +0000

Outage 2024-12-19 11:30:00 UTC - Brew service is down. All testing relying on Brew are affected. See https://brewweb.engineering.redhat.com/brew/

[Resolved] Testing Farm fails to authenticate to OpenStack

Tue, 10 Dec 2024 08:00:00 +0000

Outage 2024-12-10 15:00:00 UTC - Resolved by using new credentials for Openstack.

Outage 2024-12-10 08:00:00 UTC - Testing Farm fails to authenticate to OpenStack, we are investigating the root cause.

[Resolved] Resilience Testing outage

Fri, 06 Dec 2024 12:30:00 +0000

Resolved 2024-12-06 19:00:00 UTC - The terting is finished.

Outage 2024-12-06 12:30:00 UTC - Due to Resilience Testinga lot of services, such as brew, mbs, dit-git are down. The planned resolvance time is 2024-12-06 19:00:00

[Resolved] artifacts.osci.redhat.com outage

Thu, 05 Dec 2024 13:00:00 +0000

Resolved 2024-12-05 15:30:00 UTC - The outage was resolved. The issue was that IT falsly shut down the instance as unclaimed. The running jobs could be affected and never end (end up in running state). Please restart your testing if needed. New jobs are not affected from now.

Outage 2024-12-05 13:00:00 UTC - Archiving of artifacts in Red Hat ranch is affected due to outage artifacts.osci.redhat.com.

[Resolved] RHEL-9.6.0-image-mode is broken

Tue, 26 Nov 2024 00:00:00 +0000

Resolved 2024-12-05 15:30 UTC - RHEL-9.6.0-image mode is available on Testing Farm and tmt again. Virtualization support and aarch64 is still in works, but outside of that it should be ready for consumption.

Outage 2024-11-26 00:00 UTC - The image mode MVP testing on RHEL-9.6.0 is currently blocked, the image has expired, we are working this out.

[Resolved] download.devel SSL certificate expired

Sun, 24 Nov 2024 21:00:00 +0000

Resolved 2024-11-25 10:30 UTC - Outage was resolved by IT. Certificate was refreshed.

Outage 2024-11-24 23:00 UTC - See https://redhat.service-now.com/surl.do?n=INC3256074 for details.

Outage 2024-11-24 21:00 UTC - The download.devel SSL certificate has expired, this is affecting BaseOS CI and Testing Farm Red Hat ranch.

[Resolved] changed script for postinstall broke jobs in beaker

Tue, 19 Nov 2024 17:00:00 +0000

Resolved 2024-11-21 07:30 UTC - revert deployed, restart affected jobs if needed

Outage 2024-11-21 07:00 UTC - recent change broke jobs running in beaker

[Resolved] RHEL 10 CTC outage on Red Hat Ranch - broken compose

Tue, 19 Nov 2024 00:00:00 +0000

Resoled 2024-12-04 02:00 UTC - few tooling fixes applied and compose added also as RHEL-10.0-ctc and RHEL-10.0-ctc1

Updata 2024-11-21 00:00 UTC - we already have new compose without this issue: RHEL-10.0-20241120.1, not yet promoted to CTC

Outage 2024-11-19 00:00 UTC - RHEL 10 CTC testing affected on Red Hat Ranch due to broken compose (https://issues.redhat.com/browse/RHEL-66937).

[Resolved] Red Hat Ranch / BaseOS CI Outage - provisionig issues during the weekend

Sat, 09 Nov 2024 00:00:00 +0000

Resolved 2024-11-11 10:00 UTC - Queues are consumed, outage is over.

Outage 2024-11-11 09:00 UTC - We are working on resolving the issue in about an hour.

Outage 2024-11-10 00:00 UTC - Requests are blocked and the queue is rising. All requests will timeout out and restarting will be needed after the weekend.

Outage 2024-11-09 00:00 UTC - Due a bug causing the Artemis worker tasks to wait forever to extend reservations in PEK labs, our provisioner went down.

[Resolved] Testing Farm 2024-10.1 released

Thu, 07 Nov 2024 08:00:00 +0000

Resolved 2024-11-11 08:00 UTC - Everything looks ok, end of outage

Update 2024-11-11 09:00 UTC - Release should be stable, the other outages are not related to the release itself. Anyway, keeping this here as a note for one more week to raise awareness in case somebody runs into issues.

Update 2024-11-07 18:30 UTC - We should be stable now and deployed to 2024-10.1. There can be an issue where the results viewer is not loading some older results, no data gone, just it does nto present the data and you get an empty screen, We are trying to resolve it.

Update 2024-11-07 09:30 UTC - A bit of queue can happen. Please be patient.

Update 2024-11-07 09:00 UTC - Note that the new release is introducing some breaking changes, see a preview in the https://gitlab.com/testing-farm/docs/root/-/merge_requests/153[release notes draft].

Update 2024-11-07 08:10 UTC - A short outage caused some jobs to error out on Public Ranch, please restart your jobs.

Info 2024-11-07 08:00 UTC - We are deploying Testing Farm 2024-10.1. We are trying to avoid outages, but if you see something, check this status.

[Resolved] Testing Farm API down

Wed, 06 Nov 2024 14:30:00 +0000

Resolved 2024-11-06 15:10 UTC - The outage is over. API is back. Please restart your jobs if you were affected. We are sorry for the inconvenience.

Outage 2024-11-06 14:30 UTC - We are suffering from an API outage, jobs will break, can be stuck forever.

[Resolved] Fedora 41 outage on Red Hat ranch

Mon, 04 Nov 2024 18:00:00 +0000

Resolved 2024-11-04 13:00 UTC - The issue is resolved, enjoy!

Update 2024-11-04 13:00 UTC - The issue was identified, seems dnf5 broke the environment, and we are working on restoring the support. ETA the next day.

Outage 2024-11-02 13:00 UTC - Fedora 41 does not provision on Red Hat ranch. We are investigating the issue.

[Resolved] Gitlab CEE outage affecting jobs run on BaseOS CI and Red Hat ranch.

Mon, 04 Nov 2024 18:00:00 +0000

Resolved 2024-11-05 03:00 UTC - The outage is over. See outage list for more details.

Outage 2024-11-04 18:00 UTC - GitLab CEE outage affecting a lot of jobs.

[Resolved] BaseOS CI and Red Hat Ranch affected by provisioning outages. BaseOS CI down.

Fri, 01 Nov 2024 00:00:00 +0000

Resolved 2024-11-01 09:00 UTC - Outage was resolved. Please restart jobs if ended in error state.

Outage 2024-11-01 00:00 UTC - BaseOS Jenkins is down, and Red Hat ranch provisioning ending in error state.

[Resolved] BaseOS CI and Red Hat Ranch affected by OpenShift outage

Thu, 31 Oct 2024 12:00:00 +0000

Partial outage 2024-10-31 12:00 UTC - BaseOS CI and Red Hat Ranch affected by maintenance of RDU2 OpenShift.

[Resolved] Red Hat Ranch / BaseOS CI Outage - provisionig issues

Tue, 03 Sep 2024 20:00:00 +0000

Outage 2024-09-04 02:00 UTC - We were not able to resolve the issues during night, your requests might error out or get canceled.

Outage 2024-09-03 20:00 UTC - We are experiencing provisioning issues on Red Hat ranch.

[Resolved] Red Hat Ranch Outage - large logs consuming all the space. Testing-Farm dispatcher is disabled

Fri, 30 Aug 2024 08:30:00 +0000

Outage 2024-08-30 13:30 UTC - issue was identified, outage is resolved, please retru your requests if you they failed unexpectedly or did not finish

Outage 2024-08-30 08:30 UTC - dispatching new jobs is disabled, investigating extensive logging which caused full disk

[Resolved] Red Hat Ranch and BaseOS-CI Outage - dns misconfiguration, some crucial sites not accessible

Tue, 13 Aug 2024 15:40:00 +0000

Resolved 2024-08-14 00:50 UTC - both services fully operational.

Update 2024-08-13 23:42 UTC - dns issues looks resolved, baseosci resumed (long queque), testing farm under testing

Outage 2024-08-13 15:50 UTC - because of issue with dns servers, some internal sites are not accessible, BaseOS-CI is suspended to lower impact

[Resolved] Testing Farm API outage and longer queues on both ranches.

Thu, 08 Aug 2024 04:00:00 +0000

Outage 2024-08-08 06:15 UTC - We resumed normal operation. Outage is over.

Outage 2024-08-08 04:00 UTC - We experienced an api outage. If you encounted some issues, please restart your requests.

[Resolved] Beaker PPC64 Outage - large queue because of limited hardware

Fri, 26 Jul 2024 00:00:00 +0000

Resolved 2024-08-02 00:00 UTC - ppc64le queue times stayed below 4 hours for 3 days.

Outage 2024-07-26 00:00 UTC - due to a deteriorating PPC64 pools in Beaker, testing on this ARCH is not advised. Expect over 24 hours delay.

[Resolved] Public and Redhat Ranch Outage - broken deployment

Thu, 25 Jul 2024 13:00:00 +0000

Update 19:45 UTC - almost all jobs from queue has been processed.

Update 18:18 UTC - deployemnt fixed, queues are being processed but it will take some time.

Outage 13:00 UTC - Issue with deployment affecting all new jobs.

[Resolved] Public Ranch - Large queue

Mon, 17 Jun 2024 13:00:00 +0000

Warning 2024-06-17 13:00 UTC - Public ranch has 1k+ queued jobs, seems Fedora CI load, be patient, it is getting progressed, but expect delays.

[Resolved] Red Hat Ranch and BaseOS-CI Outage - broken krb infra

Tue, 28 May 2024 18:00:00 +0000

Resoled 2024-05-29 10:00 UTC - Change related to krb infra reverted, everything works now.

Update 2024-05-29 9:00 UTC - BaseOS-CI partially unblocked by deploying workaround (missing fix in artemis)

Update 2024-05-29 10:00 UTC - something has changed in kerberos infra leading to broken keytab. All requests to beaker are affected

[Resolved] Red Hat Ranch Outage - large queue, jobs getting delayed

Mon, 27 May 2024 00:00:00 +0000

Resolved 2024-05-27 12:00 UTC - the problem had been resolved, resolving

Update 2024-05-27 00:00 UTC - we are investigating the problem, no eta yet, problem started most probably earlier

[Resolved] Red Hat Ranch Outage - large queue, jobs getting delayed

Sun, 19 May 2024 06:00:00 +0000

Resolved 2024-05-20 20:00 UTC - things should be stable, plus rpmdeplint and rpminspect should work again

Update 2024-05-20 11:00 UTC - we got into some weird problem with provisioner, causing workers to get stuck, seems we were able to get provisioner back into a working state. We will soon start enabling back the workers.

Update 2024-05-20 08:00 UTC - we are investigating the problem, no eta yet

Outage 2024-05-19 06:00 UTC - large queue started to appear during the weekend

[Resolved] Red Hat Ranch Degraded

Thu, 28 Mar 2024 00:00:00 +0000

Resolved 22:30 UTC - Beaker is back. Degradation should be over.

Degraded 00:00 UTC - Red Hat Ranch is degraded due to Beaker planned outage (migration to AWS). Jobs for ppc64le, s390x and specific HW requirements satisfied only by Beaker will fail.

[Resolved] Long queues in Public and Red Hat ranch

Tue, 12 Mar 2024 15:00:00 +0000

Resolved 20:00 UTC (next day) - The problems should be gone. If you see weird failures please restart, unfortunately we had outage with artifacts storage, so rsync might have timed out. Should be all ok now.

Outage 15:00 UTC - Long queues detected, due to workaround gone bad. We are scaling and hopefully the issues should be over. Please restart if you run into some issues. Testing should be otherwise stable.

[Resolved] Testing Farm Red Hat Ranch outage during the weekend

Sun, 19 Nov 2023 06:00:00 +0000

Resolved 2023-11-20 09:00 UTC - We do not know the root cause, but we will try to improve the liveness probes to autoheal the deployment if the problem happens. Tracked under https://issues.redhat.com/browse/TFT-2353[TFT-2353].

Update 2023-11-20 06:00 UTC - The provisioner is working again, we are trying to find out what happened and how to deal with the problem automatically.

Outage 2023-11-19 06:00 UTC - Our provisioner has stopped processing new requests during the weekend. We are investigating the issue.

[Resolved] BaseOS CI and Testing Farm Red Hat Ranch outage due to Beaker unavailability

Tue, 07 Nov 2023 15:00:00 +0000

Resolved 19:00 UTC - Beaker is back, we resumed BaseOS CI and Testing Farm, jobs are processing.

Outage 15:00 UTC - Beaker is down including required repositories which break all executions on RH ranch for RHEL. We disabled the ranch completely to overcome too many failures. Will be posting updates here.

[Resolved] Testing Farm Red Hat ranch and BaseOS CI outage for RHEL9 and RHEL-8.1.0

Mon, 25 Sep 2023 00:00:00 +0000

Resolved 16:30 UTC - All affected composes updated, the problems should be gone. We will try to auto-restart fail jobs, please feel free to restart yourself also if in hurry.

Update 12:30 UTC - All problems affecting building had been fixed. We are getting fresh composes through. ETA 2-3 hours.

Outage 0:00 UTC - Composes have expired due to a sad misunderstanding from Friday :( We are trying to resolve the issue, but it will take few hours.

[Resolved] Console logs are disabled due to a discovered bug

Mon, 18 Sep 2023 14:00:00 +0000

Resolvede 2023-09-19 21:00 UTC

Issue was resolved in Artemis: https://gitlab.com/testing-farm/artemis/-/merge_requests/1160

Outage 2023-09-18 8:00 UTC

We have found a bug we need to address and had to temporarily disable the console log functionality. ETA 1-2 days.

[Resolved] Red Hat Ranch outage during weekend

Sun, 17 Sep 2023 08:00:00 +0000

Outage 16:00 UTC - Resolved.

Outage 8:00 UTC - We had an outage during the weekend in our provisioner. Some jobs might have failed. Please restart your testing if needed.

[Resolved] Console log has wrong link in results viewer or not available at all.

Fri, 15 Sep 2023 08:00:00 +0000

Resolved 2023-09-26 18:00 UTC

Console log links are now correct with a hotfix. Resolving outage.

Outage 2023-09-15 8:00 UTC

Red Hat ranch: Console log has wrong link in the results viewer for Red Hat ranch. We are looking into the problem. Until then, please remove the work-.* path from the URL. For example http://artifacts.osci.redhat.com/testing-farm/10de2a43-a2a1-4c4f-ab57-cf5e4416c93b/work-create-osbuild1e5bpd1y/console-94d0f6af-27a9-416c-9412-32bcf440dcce.log should be http://artifacts.osci.redhat.com/testing-farm/10de2a43-a2a1-4c4f-ab 57-cf5e4416c93b/console-94d0f6af-27a9-416c-9412-32bcf440dcce.log, note the work-create-osbuild1e5bpd1y path removed in this case to get a valid link.

Public ranch: Console logs are not yet working on Public ranch, we are working to get them available also.

[Resolved] Composes not updating on Red Hat ranch

Mon, 11 Sep 2023 08:00:00 +0000

Update 2023-09-25 12:00 UTC - Composes are being updated now.

Update 2023-09-18 8:00 UTC - Seems we finally have a a way forward here. We hope to fix the issue today finally.

Outage 2023-09-11 8:00 UTC - Composes are not updating properly due to issues with the build process. We are trying to fix the issue.

[Resolved] Beaker testing blocked on Red Hat Ranch and BaseOS CI

Mon, 21 Aug 2023 08:00:00 +0000

Resolved 12:00 UTC - The issue was resolved by Beaker admins. Outage is resolved.

Update 9:00 UTC - Seems the issue is only with /distribution/reservesys, the task was disabled in Beaker :) We are waiting for admins to resolve.

Outage 8:00 UTC - All testing against Beaker is blocked due to missing /distribution tests from Beaker. Investigation is ongoing.

[Resolved] Large queue times on Red Hat Ranch

Mon, 07 Aug 2023 00:00:00 +0000

Update 7:22 UTC - TF queue has been consumed, operating as usual.

Investigating 7:22 UTC - We’re aware of an high number of requests affecting queue times on Red Hat Ranch of Testing Farm. Testing can be delayed several hours.

[Resolved] Red Hat Ranch affected by large queue

Thu, 27 Jul 2023 09:00:00 +0000

Resolved - Queue reduced from > 600 to 15

Outage - Adding additional 8 workers.

Outage - ETA 1hr

Outage - Queue is reducing

Outage - Added additional 8 workers.

Outage - Testing jobs may time out due to high load.

Outage - Added additional 8 workers.

Outage - Testing jobs have timed out due to high load.

[Resolved] Red Hat Ranch affected by RH infra outage

Thu, 01 Jun 2023 10:30:00 +0000

Resolved - Outage was resolved, all affected builds restarted.

Outage - Testing might fail due to internal RH infrastructure having a planned outage. ETA 1 day.

[Resolved] Unexpected outage due to new release 2023-04.1

Fri, 05 May 2023 02:00:00 +0000

Outage 2023-05-05 12:00 UTC - Outage was resolved.

Outage 2023-05-05 02:00 UTC - We are deploying new Testing Farm, and it is not going super well. Expect delays and in case of unexpected errors, please restart your jobs.

[Resolved] Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Tue, 02 May 2023 08:00:00 +0000

Resolved 2023-05-02 12:00 UTC - Problem was resolved. Queue was resolved, requests should start resuming, no issues are expected.

Outage 2023-05-02 08:00 UTC - We stopped BaseOS CI and Testing Farm Red Hat ranch, because we see Artemis provisioning c5.24xlarge machines, we are investigation why. This is causing elevated costs.

[Resolved] Red Hat Ranch - deploying 2023-03.1 release - large queue expected

Thu, 06 Apr 2023 10:00:00 +0000

Notice 2023-04-06 14:00 UTC - Testing Farm 2023-03.1 was deployed to Red Hat ranch \o/

Notice 2023-04-06 10:00 UTC - we deploying Testing Farm 2023-03.1 to Red Hat ranch, larger queue is expected.

[Resolved] Red Hat Ranch - provisioning outage

Mon, 03 Apr 2023 08:00:00 +0000

Outage 2023-04-03 12:00 UTC - we believe the issue was resolved, please restart your jobs if needed.

Outage 2023-04-03 08:00 UTC - we are experiencing outage of our provisioner. It is expected some requests will end in error. Also we have a larger queue, so longer test times expected.

[Resolved] Fedora-Rawhide broken in Red Hat Ranch

Fri, 10 Mar 2023 08:00:00 +0000

Resolved 2023-03-16 12:00 UTC - We have found the issue, Rawhide is broken on Xen instances. Filled to Fedora infrastructure: https://pagure.io/fedora-infrastructure/issue/11185. We workarounded by using nitro instances for the testing.

Outage 2023-03-10 08:00 UTC - Fedora Rawhide is broken in Red Hat ranch and BaseOS CI. The instance does not boot. Requests will error out. Currently the root cause is unknown and not ETA for resolving.

[Resolved] Red Hat internal infrastructure outage

Fri, 10 Feb 2023 09:00:00 +0000

Outage 18:00 UTC 11th Feb - All processing resumed, long queue for next few hours.

Outage 16:00 UTC - Requests to Red Hat ranch might fail unexpectedly due to the internal outage. The RHEL images depend on some of the infrastructure being down - namely repositories to beakerlib and such. The outage should last until Fri, 10 Feb 2023 22:00:00 UTC.

Outage 09:00 UTC - We paused processing of all jobs via BaseOS CI, to mitigate outage effects.

[Resolved] Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Wed, 30 Nov 2022 09:00:00 +0000

Resolved 16:40 UTC - Outage was resolved. There were multiple issues here - incorrect DNS settings, code sensitive on DNS resolution, podman not using host networking.

Update 14:40 UTC - We are testing a woraround to be able to enable back the service. Fingers crossed.

Update 13:40 UTC - Seems we found a setup which makes the DNS resolving more reliable (removing aadvark-dns) and a bug from the latest release, making this error not being retried. As for why it worked before ….

Reopened 11:45 UTC - The IT DNS was a false lead. Seems we are hitting some podman DNS bug. Investigation continues.

Resolved 11:45 UTC - As it turned out, the VPC we used for years was using a DNS setup that can be prone to problems. After migrating to another VPC the problem is gone.

Update 10:30 UTC - The problems have been identified, currently it seems IT DNS problems causing random resolution errors. We are testing a workaround and working with IT to resolve the problem.

Outage 8:00 UTC - We are investigating some DNS problems on the workers. All tests might fail with Failed to establish a new connection: [Errno -2] Name or service not known'.

[Resolved] Red Hat Ranch outage - artifacts storage empty

Tue, 29 Nov 2022 08:00:00 +0000

Resolved 10:30 UTC - The issue was addressed via https://gitlab.com/testing-farm/infrastructure/-/merge_requests/102. Note that all affected jobs wil need a restart.

Update 9:00 UTC - We have identified the problem and are working on a fix.

Outage 8:00 UTC - The artifacts.osci.redhat.com is empty after run finished executing. This is due to a bad change in the artifacts syncing code. WE are working on a fix.

[Resolved] Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Mon, 10 Oct 2022 08:00:00 +0000

Outage - Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch is disabled. Currently there is no ETA for resolving the problem.

[Resolved] rpminspect fails with "internal libpod error"

Tue, 08 Feb 2022 12:00:00 +0000

Resolved 2022-04-09 - Issue was resolved by updating to a podman pre-release which is fixing the bug.

Update 2022-03-25 - Upstream has a fix for the problem. We are waiting for a custom module build with the fix, and we will update our workers with it.

Update 2022-02-25 - We have found a minimal reproducer - https://github.com/containers/podman/issues/13227#issuecomment-1050997458

Update 2022-02-25 - Investigation is still ongoing, we are not able to reproduce the problem manually. We will try today to update to podman 4.0.1 to see if it helps with the issue.

Update 2022-02-14 - Issue was filled to podman - https://github.com/containers/podman/issues/13227

Investigating - We’re aware of an issue affecting rpminspect and causing some of the tests blow up on errors like “Error: timed out waiting for file /var/lib/containers/storage/overlay-containers/4dbddac31d1a5529d055922e3fca8d200585b5ca562e7eb9976f32d7dd96b8f2/userdata/1e05be2210ea6433b84bba70c720b55586d4987fde7005428492e583d8ebc3a7/exit/4dbddac31d1a5529d055922e3fca8d200585b5ca562e7eb9976f32d7dd96b8f2: internal libpod error”

[Resolved] Large queue times on Red Hat Ranch

Thu, 27 Jan 2022 10:30:00 +0000

Update 2022-02-25 - Testing Farm was scaled and should have enough capacity for the current workload.

Update 2022-02-01 - Currently the load is ok, but we are still working on a more permanent solution

Update - We are working on a fix to scale the ranch to handle the load.

Investigating - We’re aware of an issue affecting queue times on Red Hat Ranch of Testing Farm. Testing can be delayed several hours. It is mostly affecting rpmdeplint and rpminspect.

[Resolved] Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Wed, 29 Dec 2021 11:00:00 +0000

Update - Images were updated, failed tests restarted. In case you encounter issues, please restart your failed jobs.

Update - Red Hat tracking issue

Investigating - We’re aware of expired images for RHEL-8.6.0 and RHEL-9.0.0. They are causing failures for installability test and all functional tests. We are working to resolve the problem ASAP and will try to restart all affected jobs once the issue is resolved.

[Resolved] Issues with provisioning in Red Hat Ranch

Tue, 21 Dec 2021 16:00:00 +0000

Update - Issue was reported to Artemis, we will continue investigation there.

Update - We resolved the problem by restarting the worker. We are preparing an issue to Artemis to follow up. Looks like in this case the worker was not able to reconnect to the rabbitmq for whatever reason. The outage is cleared, one more update to the issue will follow once we report the problem.

Investigating - We’re aware of an issue affecting provisioning of machines on Red Hat Ranch of Testing Farm. Seems the problem is with our Artemis provisioning. We will be updating this issue once we know more. All queues are paused, no requests should be lost for now.