Testing Farm Status

[Resolved] CTS outage affecting Testing Farm, expect queue

Mon, 01 Jun 2026 09:55:00 +0000

Outage | 2026-06-01 09:55:00 UTC

Outage of CTS is affecting Testing Farm Beaker testing. We disabled queue processing and hoping for a quick fix. Expect delays.

Update | 2026-06-01 10:39:00 UTC

The issue is resolved, we are confirming and resuming the queue.

Resolved | 2026-06-01 10:45:00 UTC

Resolved. CTS was brought back to life.

[Resolved] quay.io outage

Mon, 01 Jun 2026 08:10:00 +0000

Outage | 2026-06-01 08:10:00 UTC

We are tracking a quay.io outage not yet announced on status.redhat.com. This might affect some testing especially if containers involved.

Update | 2026-06-01 08:42:00 UTC

The outage is now tracked on RH status: https://status.redhat.com/incidents/jn6r256zc62c

Resolved | 2026-06-01 11:15:00 UTC

Outage was resolved, quay is back

Reopened | 2026-06-01 14:20:00 UTC

Outage was reopened, quay is down again

Resolved | 2026-06-01 22:23:00 UTC

Outage looks cleared. Resolving. The quay team is still monitoring, but from our end everything looks stable now.

[Resolved] Artifacts storage for RH ranch outage

Sat, 30 May 2026 14:30:00 +0000

Outage | 2026-05-30 14:30:00 UTC

There was an outage of artifacts storage for 5 hours during weekend, and for 40 minutes Monday morning. If you see artifacts unavailable (403) please restart your requests / CI jobs.

Update | 2026-06-01 08:16:00 UTC

The issue should be resolve now. Keeping open for few hours for visibility.

Resolved | 2026-06-03 00:48:00 UTC

Outage resolved

[Resolved] Red Hat ranch dispatching blocked.

Wed, 27 May 2026 14:05:00 +0000

Outage | 2026-05-27 14:05:00 UTC

WE are investigating an issue with nomad dispatching. Expect queue.

Update | 2026-05-27 16:59:00 UTC

The problem is gone, we are back to normal operation. Dispatcher should now be more capable in dispatching jobs speeding things up.

Resolved | 2026-05-27 17:06:00 UTC

Resolved

[Resolved] Large load on RH ranch, we are scaling up

Tue, 26 May 2026 13:30:00 +0000

Outage | 2026-05-26 13:30:00 UTC

We are seeing larger load on Red Hat ranch, we are scaling up

Resolved | 2026-05-27 15:15:00 UTC

The outage is cleared from yesterday. Queue gone.

[Resolved] Higher error rate on Testing Farm Red Hat ranch

Mon, 25 May 2026 15:30:00 +0000

Outage | 2026-05-25 15:30:00 UTC

We are investigating higher error rate on Testing Farm RH ranch

Update | 2026-05-26 03:36:00 UTC

There was an issue with SSL certs on download location affecting the testing. The problem seems to be resolved now.

Resolved | 2026-05-26 03:36:00 UTC

resolving

[Resolved] Fedora Rawhide provisioning often fails due to service ordering bug

Mon, 25 May 2026 07:00:00 +0000

Outage | 2026-05-25 07:00:00 UTC

We are investigating an issue with Fedora Rawhide causing the machine to not boot up on ordering isssues. Investigation is currently in progress to find which component caused this issue.

Update | 2026-05-25 22:38:00 UTC

Seems a systemd-261 bug: https://bugzilla.redhat.com/show_bug.cgi?id=2481304

Update | 2026-05-26 03:37:00 UTC

Isssue might affect also Fedora-eln very soon.

Update | 2026-05-28 08:44:00 UTC

The fixed systemd should be now in rawhide repos, checking if this is resolved.

Resolved | 2026-05-28 14:00:00 UTC

Resolving, this problem should be gone now.

[Resolved] Large queue on Red Hat ranch

Wed, 20 May 2026 08:30:00 +0000

Outage | 2026-05-20 08:30:00 UTC

We are aware of large queue on testing farm and are looking into it. Sorry for the inconvenience.

Resolved | 2026-05-20 22:20:00 UTC

Queue is gone.

[Resolved] Composes not update in Testing Farm

Wed, 20 May 2026 07:00:00 +0000

Outage | 2026-05-20 07:00:00 UTC

Unfortunately ITUP is broken and our Jenkins is down hosting the compose-update process. No ETA for now :(

Update | 2026-05-21 22:46:00 UTC

Jenkins is back, we should see fresh compose update during Friday.

Update | 2026-05-24 23:45:00 UTC

Jenkins is back, but the composes had not been yet updated, we are working it out.

Update | 2026-05-25 06:38:00 UTC

Compose update is in progress, next update in 2 hours

Resolved | 2026-05-26 03:37:00 UTC

The issue was resolved. Composes are now updated.

[Resolved] Rpminpsect on Fedora-CI PRs are failing when trying to download a scratch-build

Fri, 15 May 2026 08:39:00 +0000

Outage | 2026-05-15 08:39:00 UTC

https://forge.fedoraproject.org/ci/tickets/issues/560

Resolved | 2026-05-18 22:00:00 UTC

Rpminspect image updated

[Resolved] Red Hat ranch requests are broken due to Artemis

Mon, 04 May 2026 12:30:00 +0000

Outage | 2026-05-04 12:30:00 UTC

Our provisioner went down. We are investigating why. RH ranch queue was suspended but a lot of jobs ended in error state.

Resolved | 2026-05-05 06:20:00 UTC

The outage is resolved. Seems everything is back to normal now.

[Resolved] BaseOS CI is not triggering

Thu, 30 Apr 2026 11:57:00 +0000

Outage | 2026-04-30 11:57:00 UTC

Due to various issue the triggering of BaseOS CI broke down. We are working on resuming the triggering and will be restarting from 30th April to deal with the outage.

Update | 2026-05-04 15:34:00 UTC

Triggering whould be working now. We are working on rerun all the missed triggers.

Resolved | 2026-05-12 12:15:00 UTC

This issue was resolved, all affected jobs restarted.

[Resolved] Large queue on public ranch

Wed, 22 Apr 2026 10:17:00 +0000

Outage | 2026-04-22 10:17:00 UTC

The queue on public ranch has exploded, we are trying to scale up.

Update | 2026-04-22 23:44:00 UTC

Queues are back to normal. We were able to scale up, so we should have a bit more firepower here.

Resolved | 2026-04-22 23:45:00 UTC

Resolving

[Resolved] Elevated API errors

Mon, 20 Apr 2026 15:19:00 +0000

Outage | 2026-04-20 15:19:00 UTC

We are seeing elevated API errors, we are investigating the issue.

Update | 2026-04-20 16:14:00 UTC

This should be better now.

Resolved | 2026-04-20 17:52:00 UTC

Resolving, the issue is gone

[Resolved] Testing Farm RH ranch not processing job from April 18th

Mon, 20 Apr 2026 06:38:00 +0000

Outage | 2026-04-20 06:38:00 UTC

Unfortunatley we forgot to rotate one of th tokens and the RH ranch is not processing any jobs from Saturday.

Update | 2026-04-20 08:31:00 UTC

This is now resolved. We canceld all hummingbird requests, the team will restart them. Others except java testing are restarted in place.

Resolved | 2026-04-20 15:18:00 UTC

This is resolved now

[Resolved] Errors when regenerating tokens via the UI

Thu, 16 Apr 2026 11:37:00 +0000

Outage | 2026-04-16 11:37:00 UTC

We are aware of an issue when regenerating tokens, we are looking into the issue.

Resolved | 2026-04-20 16:15:00 UTC

This issue should be resolved for some time now

[Resolved] Outdated image mode

Thu, 16 Apr 2026 09:38:00 +0000

Outage | 2026-04-16 09:38:00 UTC

Base images for image mode are delayed, leading to skipped update in testing farm.

Resolved | 2026-05-05 08:24:00 UTC

This should be resolved now.

[Resolved] Fedora Rawhide and ELN composes provisioning errors out

Wed, 08 Apr 2026 14:53:00 +0000

Outage | 2026-04-08 14:53:00 UTC

Testing-farm jobs using the Fedora-ELN and Rawhide composes fail to provision or fail during the excution of the pre-artifact-installation playbook.

As far as we can tell it goes through something like: image booted, testing farm connected to it, started running playbooks, got to the tasks which upgrades all the rpms and reboots, could not connect back due to emergency mode.

Update | 2026-04-17 01:45:00 UTC

Also hitting Fedora-Rawhide now. Seems like some bug from Rawhide, investigation is ongoing.

Resolved | 2026-05-08 02:54:00 UTC

This is resolved via workarounds. Fedora-eln is now not updated to latest bits.

[Resolved] Testing Farm container based testing is broken

Wed, 25 Mar 2026 15:00:00 +0000

Outage | 2026-03-25 15:00:00 UTC

We are sorry, the rotation of secrets broken our quay.io creds.

Resolved | 2026-03-26 08:57:00 UTC

Resolved. Testing Farm container testing is back.

[Resolved] All Beaker pools are disabled due to Beaker Scheduled Maintenance

Tue, 24 Mar 2026 13:09:00 +0000

Outage | 2026-03-24 13:09:00 UTC

All Beaker pools are disabled. Estimated time - 4 hours.

Resolved | 2026-03-24 16:00:00 UTC

The maintenance has ended

[Resolved] Testing Farm 2026-03.2 being deployed

Thu, 19 Mar 2026 19:18:00 +0000

Notice | 2026-03-19 19:18:00 UTC

We are deploying new Testing Farm version with tmt-1.69, stay tuned. Small downtime of API is expected, otherwise no other issues should arise. We will update this status entry if needed. ETA 4 hours.

Update | 2026-03-20 16:02:00 UTC

There was a small bug failing ~20 jobs in RH ranch causing Ansible to fail with: The conditional check ‘ansible_pkg_mgr in [‘unknown’, ‘dnf’]’ failed

Update | 2026-03-21 22:40:00 UTC

Testing Farm is now deployed, we will send out the release notes on Monday.

Update | 2026-03-21 22:42:00 UTC

There was an issue in Ansible setup for Fedora in Red Hat ranch, some of the jobs have failed, please restart them if you encountered Could not import the libdnf5 python module using /var/opt/standalone-python/bin/python3.

Resolved | 2026-03-23 19:02:00 UTC

Testing Farm was deployed! Enjoy!

[Resolved] ibmcloud-ppc64le cloud disabled

Mon, 16 Mar 2026 11:00:00 +0000

Outage | 2026-03-16 11:00:00 UTC

Issues with error instances that can’t be removed and new instance provisioning, investigation in progress. Pool is disabled.

Resolved | 2026-03-27 11:00:00 UTC

Error instances cleaned up, pool re-reenabled with 30 instances limit.

[Resolved] Beaker's disrupting Red Hat ranch.

Thu, 12 Mar 2026 16:07:00 +0000

Outage | 2026-03-12 16:07:00 UTC

The Beaker scheduler have issues which affects Red Hat ranch’s Beaker pool. Expect delays in guest provisioning.

Resolved | 2026-03-12 19:30:00 UTC

Seems the Beaker scheduler is back and running. The incident is resolved.

[Resolved] Koji outage affects public ranch

Thu, 12 Mar 2026 11:13:00 +0000

Outage | 2026-03-12 11:13:00 UTC

Koji has an outage which affects test reqeuest with Koji or Copr artifacts. The affected requests end up with Test environment installation failed: Install packages error.

Resolved | 2026-03-12 15:39:00 UTC

Seems the koji is back and running. The incident is resolved.

[Resolved] Testing Farm outage - Cannot execute the rule

Thu, 05 Mar 2026 21:00:00 +0000

Outage | 2026-03-05 21:00:00 UTC

Both ranches are suffering outage after the most recent release deployment. A mistake in configuration is causing pipelines to fail. We are currently investigating why the corresponding fix have not been deployed properly.

Update | 2026-03-06 10:33:00 UTC

It was affecting Public Ranch too. Please restart your failed requests. The issue was fixed by reverting change. The issue was not affecting requests testing against containers.

Resolved | 2026-03-06 10:00:00 UTC

Hotfix deployed and new request are working.

[Resolved] IBM Cloud ppc64le outage

Mon, 02 Mar 2026 09:06:00 +0000

Outage | 2026-03-02 09:06:00 UTC

IBM cloud ppc64le provisioning is broken. All the ppc64le workloads from Testing Farm will land in Beaker. If you can postpone or limit your ppc64le testing via Testing Farm and BaseOS CI for testing unrelated to RC baseline testing, please do so.

Update | 2026-03-03 17:11:00 UTC

So the ppc64le queue is not exploding, we were able to unblock some ppc64le machines and seems IBM cloud is getting back. More update in few hours.

Update | 2026-03-05 11:01:00 UTC

IBM Cloud ppc64le is enabled back. Beaker ppc64le queue times are back to normal.

Update | 2026-03-05 12:09:00 UTC

Resolving.

Resolved | 2026-03-05 12:09:00 UTC

Resolved

[Resolved] RHEL-9.9.0 and RHEL-10.3 image mode testing blocked

Tue, 24 Feb 2026 14:01:00 +0000

Outage | 2026-02-24 14:01:00 UTC

The image mode testing of RHEL-9.9.0 and RHEL-10.3 is blocked on unavailability of image mode rhel container images. See https://issues.redhat.com/browse/BIFROST-1113 for details.

Resolved | 2026-03-12 16:43:00 UTC

The images are updated. The outage is closed.

[Resolved] Testing Farm 2026-02.2 was deployed on 2026-02-19

Thu, 19 Feb 2026 18:58:00 +0000

Outage | 2026-02-19 18:58:00 UTC

New Testing Farm version was deployed, but we will send out release notes on Monday, as we did not make it. For a previous of release notes see: https://testing-farm.gitlab.io/-/docs/root/-/jobs/13185451238/artifacts/preview/Testing%20Farm/0.1/releases/2026-02.2.html

Resolved | 2026-02-23 10:21:00 UTC

Resolving, new release is deployed fully.

[Resolved] RHEL-9.9, RHEL-10.1, RHEL-10.2 and RHEL-10.3 compose update blocked

Thu, 19 Feb 2026 10:22:00 +0000

Outage | 2026-02-19 10:22:00 UTC

RHEL-9.9.0, RHEL-10.1, RHEL-10.2 and RHEL-10.3 are blocked due to multiple issues. See https://issues.redhat.com/browse/TFT-4367 for details. No ETA yet, but the issues are escalated as blockers.

Resolved | 2026-02-23 23:32:00 UTC

Resolving, package mode images are now available.

[Resolved] Fedora CI jobs on bodhi updates are not triggered

Tue, 17 Feb 2026 13:10:00 +0000

Outage | 2026-02-17 13:10:00 UTC

Something is wrong on our side and the messaging bus. We are trying to get this resolved.

Resolved | 2026-02-18 14:25:00 UTC

Issue resolved, there will be a big job queue while it catches up.

[Resolved] Testing-farm jobs on rawhide fail signature check

Tue, 17 Feb 2026 12:14:00 +0000

Outage | 2026-02-17 12:14:00 UTC

Testing-farm jobs that require artifacts (for example scratch builds) are now failing on rawhide due to the signature enforcement proposal. We are investigating if this is an issue on our side or the change proposal.

For more info, follow https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BYG3JSPRB3R2YQWP6O3ZD2GMNFRVM7JS/#R2422WIXVQYIBCAXHHTWG7ITI2P6PERA

[Resolved] Large queue on public ranch due to provisioning outage

Mon, 16 Feb 2026 09:41:00 +0000

Outage | 2026-02-16 09:41:00 UTC

We just recovered from a provisioning outage on public ranch. Things should start resuming, but because if limited scalability it might take a while (1-2 hours)

Resolved | 2026-02-16 11:11:00 UTC

The queue is gone, outage resolved.

[Resolved] Some requests in Public Ranch are missing artifacts

Wed, 11 Feb 2026 00:00:00 +0000

Outage | 2026-02-11 00:00:00 UTC

We are occasionally experiencing an issue with artifacts archivation. Some requests in Public Ranch have incomplete results.

Issue is tracked in: https://issues.redhat.com/browse/TFT-4328

Resolved | 2026-03-23 20:51:00 UTC

This issue should be now resolved with 2026-03.2 release.

[Resolved] Broken Fedora repositories

Tue, 10 Feb 2026 23:00:00 +0000

Outage | 2026-02-11 00:00:00 UTC

We are experiencing outage of repositories on Fedora images.

Issue is tracked in: https://forge.fedoraproject.org/infra/tickets/issues/13138

Updating and loading repositories:
 Fedora 43 openh264 (From Cisco) - x86_ 100% |   8.6 KiB/s |  10.8 KiB |  00m01s
Failed to download metadata
 Error loading local metadata for repository "updates"
  Librepo error: Checksum error /var/cache/libdnf5/updates-431245df5652aba0/repodata/72324174538178d41d9e8bc090d9eb64666f071a34c516937f13e6932546a129-filelists.xml.zck: Unable to read zchunk lead

Resolved | 2026-02-17 01:02:00 UTC

Resolving, seems the issue was addressed and we do not see it anymore.

[Resolved] RHEL-8.10-ppc64le outage due to standalone python

Tue, 10 Feb 2026 00:00:00 +0000

Outage | 2026-02-10 00:00:00 UTC

We are experiencing outage of RHEL-8.10-ppc64le due to standalone python runtime.

Issue is tracked in: https://issues.redhat.com/browse/TFT-4325

Resolved | 2026-03-23 21:15:00 UTC

Resolved in TF 2026-03.2 release

[Resolved] Fedora 44 not available in Testing Farm

Mon, 09 Feb 2026 11:46:00 +0000

Outage | 2026-02-09 11:46:00 UTC

The import is in progress after branching. We are working it out. ETA later today.

Resolved | 2026-02-10 02:45:00 UTC

Fedora 44 is now available.

[Resolved] High queue on public ranch, expect delays

Thu, 05 Feb 2026 11:00:00 +0000

Outage | 2026-02-05 11:00:00 UTC

We are experiencing high load on Testing Farm. Investigation in place. Expect delays.

Update | 2026-02-05 15:00:00 UTC

The situation is ongoing, current average queue time is 3.5 hours

Update | 2026-02-06 01:40:00 UTC

Things are improving, we had to kill testing from scap via atex. Seems like some new integration using reservations is not cleaning up resources, causing all the requests to stay up long and take all resources, blocking other users.

Update | 2026-02-06 02:00:00 UTC

queue is almost gone, we are monitoring the situation

Resolved | 2026-02-09 10:26:00 UTC

Problems should be gone as of last week. The misusage was addressed on the user side.

[Resolved] Testing Farm 2026-02.1 deployment in progress

Mon, 02 Feb 2026 10:46:00 +0000

Outage | 2026-02-02 10:46:00 UTC

We are deploying a new TF release, brief outages of API might happen during this time.

Update | 2026-02-02 18:51:00 UTC

Deployment is half done, and ongoing to the next day

Resolved | 2026-02-04 15:59:00 UTC

Testing Farm 2026-02.1 is fully deployed. Resolving.

[Resolved] Image Mode doesn't work correctly on the redhat ranch

Thu, 29 Jan 2026 10:21:00 +0000

Outage | 2026-01-29 10:21:00 UTC

We discovered problems with image-mode composes mapping. Testing Farm may use non-image-mode images for image mode testing. We are working on a fix.

Update | 2026-02-02 18:38:00 UTC

The outage is still not resolved, we are looking into the issue.

Resolved | 2026-02-02 11:47:00 UTC

Resolved.

[Resolved] Red Hat Ranch and BaseOS CI not working due to provisioner outage

Wed, 28 Jan 2026 00:00:00 +0000

Outage | 2026-01-28 00:00:00 UTC

Provisioner does not work, blocking processing all requests on Red Hat Ranch and BaseOS CI.

Update | 2026-01-28 09:00:00 UTC

The outage was resolved, the queue is going down.

Resolved | 2026-01-28 12:00:00 UTC

The queue is now gone.

[Resolved] High load - delays and timeouts

Thu, 22 Jan 2026 00:00:00 +0000

Outage | 2026-01-22 00:00:00 UTC

We are working to cancel the problematic jobs. We apologize for the inconvenience.

Update | 2026-01-22 20:43:00 UTC

Things should be finally resuming. We were today hamerred by requests trying to install 3k+ artifacts from ELN. Unfortunately the initial clenaup did not go well and we found a bug preventing the jobs to die. That caused delays. We are sorry for the trouble. The queue should start slowly go down. We are trying to scale up to deal with the current queue.

Resolved | 2026-01-23 00:00:00 UTC

All problematic jobs removed and queue is back to 0. Thank you for you patience.

[Resolved] IBM Cloud outage

Wed, 21 Jan 2026 00:00:00 +0000

Outage | 2026-01-21 00:00:00 UTC

Due to various problems with IBM Cloud, we have had to disable it’s usage temporarily until these problems are resolved. Machines from Beaker will be used and you will notice a significant delay completing request for the architectures: s390x, ppc64le

Resolved | 2026-01-29 17:55:00 UTC

Outage was resolved. Pool enabled back.

[Resolved] BaseOS CI fails to send messages to UMB

Tue, 20 Jan 2026 12:27:00 +0000

Outage | 2026-01-20 12:27:00 UTC

BaseOS CI fails to send any UMB message. It blocks all of the testing. The issue is reported and we are monitoring the progress.

Resolved | 2026-01-20 13:56:00 UTC

The outage seems resolved. We are restarting affected builds. If we miss yours - please reach out to us.

[Resolved] Compose update blocked due to issues in IBM cloud

Tue, 20 Jan 2026 09:00:00 +0000

Outage | 2026-01-20 09:00:00 UTC

We are having issues pushing through update on IBM cloud, this is blocking all compose updates. IBM support is looking into the issue.

Resolved | 2026-02-02 10:45:00 UTC

This is now resolved.

[Resolved] Elevated API errors in Testing Farm

Mon, 19 Jan 2026 09:00:00 +0000

Outage | 2026-01-19 09:00:00 UTC

We are experiecing read timeouts on our API. Issue is under investigation.

Update | 2026-01-19 20:00:00 UTC

We are patching API to better recycle stale connections

Update | 2026-01-19 20:21:00 UTC

Issue should be addressed, we are monitoring the situation. https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 20:34:00 UTC

Things are back to stable, we hotpatched production with: https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 21:11:00 UTC

The issues is back :( we are looking further

Resolved | 2026-01-19 23:33:00 UTC

Resolving, we did nto see this anymore after applying the fixes.

[Resolved] High koji build queue for ci channel

Fri, 16 Jan 2026 13:25:00 +0000

Outage | 2026-01-16 13:25:00 UTC

Fedora mass rebuild is ongoing and it has a higher priority than the ci channel for Fedora CI scratch builds. Expect a long delay to both status updates on PRs as well as scratch builds not starting.

Resolved | 2026-01-19 09:56:00 UTC

Mass rebuild build queue passed. Scratch builds queue has returned to normal

[Resolved] UMB down

Sat, 10 Jan 2026 09:00:00 +0000

Outage | 2026-01-10 09:00:00 UTC

Issue with signing server blocked umb, now cert is fixed, but UMB is overloaded

Resolved | 2026-01-13 23:00:00 UTC

UMB is running, queue has been processed and BaseOS-CI works normaly again

[Resolved] Redoc documentation shows empty page

Thu, 08 Jan 2026 14:03:00 +0000

Outage | 2026-01-08 14:03:00 UTC

We have a dependencies problem which affects https://api.testing-farm.io/redoc page rendering it empty. Please use https://api.testing-farm.io/docs instead for now. We are working on a fix

Resolved | 2026-01-15 19:03:00 UTC

resolved with 2026-01.1 release

[Resolved] Public ranch outage

Tue, 16 Dec 2025 13:00:00 +0000

Outage | 2025-12-16 13:00:00 UTC

Unfortunately the provisioner went down, and it took a while we noticed :(

Update | 2025-12-17 03:40:00 UTC

The outage was addressed, and the provisioner is getting back into good shape, expect failed jobs and delays though.

Update | 2025-12-17 03:42:00 UTC

There is around 1.7k requests to process, will take few hours

Update | 2025-12-17 20:10:00 UTC

Things are back to normal from the morning. Resolving.

Update | 2025-12-19 03:30:00 UTC

Outage should be gone.

Resolved | 2025-12-19 03:30:00 UTC

Resolving

[Resolved] Provisioning errors due to missing s390x images

Fri, 12 Dec 2025 10:26:00 +0000

Outage | 2025-12-12 10:26:00 UTC

Our compose-update let through some composes with missing s390 images in IBM cloud, breaking provisioning. We are looking into the situation, for more information see: https://redhat-internal.slack.com/archives/C03BRN71JAF/p1765792914415559

Resolved | 2025-12-15 12:25:00 UTC

Errors resolved and images synced.

[Resolved] Public ranch requests testing against copr builds are broken

Wed, 10 Dec 2025 15:30:00 +0000

Outage | 2025-12-10 15:30:00 UTC

Copr added anubis that broke copr builds installation in Testing Farm. We are trying to get the team to revert it until we can find out how to solve the issue.

Update | 2025-12-10 20:06:00 UTC

Things look resolved. Copr team applied a fix to not put repo files under annubis.

Resolved | 2025-12-10 20:08:00 UTC

Resolved.

[Resolved] Beaker outage affecting provisioning

Tue, 09 Dec 2025 07:59:00 +0000

Outage | 2025-12-09 07:59:00 UTC

We are seeing elevated error rate when provisioning Beaker, investigation is in progress.

Update | 2025-12-09 08:40:00 UTC

Seems things look more stable now, reported to #help-beaker channel for investigation of the 500s we and users directly interacting with Beaker API were seeing: https://redhat-internal.slack.com/archives/C04KRPW1HJ4/p1765266395518069

Update | 2025-12-09 18:26:00 UTC

Resolved, it was caused by Beaker going out of disk space :(

Resolved | 2025-12-09 20:06:00 UTC

Resolving …

[Resolved] Fedora 42 is unavailable in Beaker

Wed, 03 Dec 2025 08:33:00 +0000

Outage | 2025-12-03 08:33:00 UTC

Fedora-42 and Fedora-42-Updated are unavailable in Beaker due to problems booting the system. Investigation is on-going. Thank you for your patience.

Update | 2025-12-03 15:32:00 UTC

Fedora-43 also affected.

Update | 2025-12-10 18:27:00 UTC

Seems the issue is caused by an SELinux problem, we are gathering more info, cannot yet confirm the issue is resolved.

Update | 2025-12-19 03:30:00 UTC

The issue still persists, for more information see: https://redhathub.service-now.com/hub?id=standard_ticket&table=universal_request&sys_id=ed719a372fe13e903538cd3bcfa4e319

Resolved | 2026-01-15 19:02:00 UTC

Resolved on Beaker side, see: https://redhat-internal.slack.com/archives/C04KRPW1HJ4/p1768392931745919

[Resolved] Public Ranch outage - jobs not finishing, high queue

Fri, 28 Nov 2025 02:00:00 +0000

Outage | 2025-11-28 02:00:00 UTC

Hi, Testing Farm got overloaded in the early morning and out provisioner did not make it. We are trying to get it back to a stable state. Expect jobs that do not finish or error out and new requests being queued for long time. We appologize for the inconvenience.

Update | 2025-11-29 00:53:00 UTC

Resolving, things should be back to stable now.

Resolved | 2025-12-01 08:27:00 UTC

Resolved for some time, closing.

[Resolved] Public Ranch provisioning unstable, high queue

Wed, 12 Nov 2025 13:00:00 +0000

Outage | 2025-11-12 08:00:00 UTC

We are seeing issues with provisioning in Public ranch. Investigation is in progress.

Resolved | 2025-11-16 20:47:00 UTC

Resolving, after updating to Testing Farm v0.0.82 with a fix seems provisioning is stable.

[Resolved] Testing Farm updating AWS DNS setup to support DNSSEC, failover to Openstack and Beaker

Thu, 06 Nov 2025 21:00:00 +0000

Outage | 2025-11-06 21:00:00 UTC

For more information see: https://groups.google.com/a/redhat.com/g/rhel-devel/c/-46R4axLj_s

Resolved | 2025-11-06 23:45:00 UTC

The setup is complete, unfortunately the changes did not have effect we hoped for. See https://redhat-internal.slack.com/archives/C03BRN71JAF/p1762277234611719 Slack for more information. Resolving the outage.

[Resolved] Beaker pools not working

Fri, 31 Oct 2025 09:05:00 +0000

Outage | 2025-10-31 09:05:00 UTC

Due to a breaking change in Artemis v0.0.81 the Beaker pools became broken and erroring out all the time.

Update | 2025-11-03 09:07:00 UTC

The problem was identified early Monday, and the fix is getting deployed to mitigate this problem. Next to it we are extending our monitoring to cover the problem.

Update | 2025-11-03 10:44:00 UTC

The issue is resolved, we are monitoring the problem.

Update | 2025-11-04 03:15:00 UTC

Issue should be resolved

Resolved | 2025-11-06 20:52:00 UTC

Resolving, things are stable now.

[Resolved] AWS world-wide outage affecting Testing Farm

Mon, 20 Oct 2025 09:29:00 +0000

Outage | 2025-10-20 09:29:00 UTC

The world wide AWS outage is affecting Testing Farm. We are looking how to mitigate the issue, but be prepared for delays in testing. Public Ranch is unaffected as it is hosted in us-east-2.

Update | 2025-10-20 18:00:00 UTC

Issues are persisting, things looked better, now it is worse and we had to disable AWS alltogether. Expect delays.

Resolved | 2025-10-21 01:04:00 UTC

Outage is resolved. AWS should be back to normal operations. We have enabled back the AWS pools in general routing.

[Resolved] quay.io is down

Mon, 20 Oct 2025 07:51:00 +0000

Outage | 2025-10-20 07:51:00 UTC

We are aware of quay.io outage, we are gatheting more information. This will affect our services, but the real extent is not yet known.

Update | 2025-10-20 07:58:00 UTC

Seems connected to the AWS ouatage in us-east-1: https://health.aws.amazon.com/health/status

Update | 2025-10-20 09:10:00 UTC

Outage was resolved by routing traffic to us-east-2.

Resolved | 2025-10-20 09:10:00 UTC

Resolving

[Resolved] Provisioning unstable in Testing Farm Public Ranch

Sat, 18 Oct 2025 10:00:00 +0000

Outage | 2025-10-18 10:00:00 UTC

During this time the outage started, the operation team was out on weekend.

Update | 2025-10-19 15:00:00 UTC

The outage was resolved by a Testing Farm engineering checking in during the weekend.

Update | 2025-10-20 08:06:00 UTC

Provisioning looks stable now, but keeping open for awareness. We are tracking the issue in https://issues.redhat.com/browse/TFT-3977.

Resolved | 2025-11-06 20:51:00 UTC

Resolving, we are not seeing any issues here.

[Resolved] Testing Farm 2025-09.1 was deployed

Fri, 17 Oct 2025 11:23:00 +0000

Outage | 2025-10-17 11:23:00 UTC

We are preparing release notes.

Update | 2025-11-12 12:41:00 UTC

Release notes are available:

https://docs.testing-farm.io/Testing%20Farm/0.1/releases/2025-09.1.html

Resolved | 2025-11-12 12:43:00 UTC

Resolving

[Resolved] Brew outage

Fri, 17 Oct 2025 09:00:00 +0000

Outage | 2025-10-17 09:00:00 UTC

Maintenance affecting brew, dist-git, MBS

Update | 2025-10-20 07:19:00 UTC

Brew is working again, resolving.

Resolved | 2025-10-20 09:11:00 UTC

Resolving

[Resolved] Fedora Koji Outage

Wed, 15 Oct 2025 07:30:00 +0000

Outage | 2025-10-15 07:30:00 UTC

The Fedora Koji is down. No ETA available currently.

We are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Update | 2025-10-16 00:06:00 UTC

The experience is better, but still elevated 503. Follow this issue to find out more: https://pagure.io/fedora-infrastructure/issue/12814

Resolved | 2025-11-12 12:52:00 UTC

Seems the issues are gone according to latest updates of https://pagure.io/fedora-infrastructure/issue/12814

[Resolved] Fedora Koji Outage

Tue, 14 Oct 2025 10:41:00 +0000

Outage | 2025-10-14 10:41:00 UTC

The Fedora Koji is down. No ETA available currently.

Update | 2025-10-14 10:43:00 UTC

We are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Resolved | 2025-10-14 11:28:00 UTC

We didn’t get any statement, but Koji started to work again.

[Resolved] Fedora CI not triggering except bodhi tests

Tue, 07 Oct 2025 22:00:00 +0000

Outage | 2025-10-07 22:00:00 UTC

Fedora CI triggers are broken, everything excpect bodhi functional tests won’t be run. We are trying to resolve, but we need OSCI to help. This includes rpminspect, rpmdeplint, installability and dist-git PR testing.

Update | 2025-10-08 23:25:00 UTC

Tracked internally https://redhat-internal.slack.com/archives/C04L304KVH9/p1759965536710569

Resolved | 2025-10-09 08:40:00 UTC

Issue was resolved by OSCI team.

[Resolved] Cyborg Loki is down, due to migration to S3

Thu, 02 Oct 2025 10:41:00 +0000

Outage | 2025-10-02 10:41:00 UTC

Unfortunately Loki is down for a longer time, as it went out of space and we need to migrate it to S3 backed storage. No ETA currently available.

Resolved | 2025-10-08 23:22:00 UTC

Loki is back in the game!

[Resolved] Fedora and CentOS stream compose broken due to download.bos down

Thu, 02 Oct 2025 10:40:00 +0000

Outage | 2025-10-02 10:40:00 UTC

We are aware of an issue affecting Fedora and CentOS Stream testing in Red Hat ranch. This is due to download.bos outage, we are working on a workaround.

Resolved | 2025-10-08 23:22:00 UTC

This is resolved, problem was addressed

[Resolved] BaseOS-CI issue with UMB

Sun, 28 Sep 2025 22:00:00 +0000

Outage | 2025-09-28 22:00:00 UTC

dispatchers are blockd on writing to UMB

Resolved | 2025-09-29 22:29:00 UTC

The issues had been resolved. Jobs are resuming. No outage except the delay is expected. Let us know if you would need some assitance.

[Resolved] Wow fails on connection to TCMS in BaseOS CI

Fri, 26 Sep 2025 10:54:00 +0000

Outage | 2025-09-26 10:54:00 UTC

We are aware of an issue in TCMS affecting some test jobs using workflow-tomorrow (wow). The error usually looks like [2025-09-15 13:54:42] [E] [test-scheduler-artemis] Exception raised: Failure during 'wow' execution: and the wow debug log shows http.client.IncompleteRead: IncompleteRead(48586 bytes read) at the end. This is caused by https://gitlab.cee.redhat.com/sustaining-engineering/tcms/-/issues/2 and the TCMS folks are looking into it.

Resolved | 2025-10-08 23:22:00 UTC

This was resolved, tcms is stable now.

[Resolved] BaseOS-CI jenkins controller migration

Mon, 22 Sep 2025 22:00:00 +0000

Outage | 2025-09-22 22:00:00 UTC

suspended queue, after some time remaining jobs will be canceled and restarted on new location

Resolved | 2025-09-23 01:30:00 UTC

BaseOS-CI jenkins controller migrated and fully operational.

[Resolved] Default instance for s390x has 10GiB disk size

Fri, 19 Sep 2025 09:31:00 +0000

Outage | 2025-09-19 09:31:00 UTC

Due to a misconfiguration the default disk size is 10GiB for s390x machines which do not specify the disk hardware requirement. To workaround, please specify the disk hardware requirement. In this case your request will be routed to Beaker instead of the default IBM cloud. We are working hard to resolve this issue, but it might take some significant time as we need to adjust our Openshift deployment due to hitting maximum configmap size limit.

Update | 2025-09-22 10:46:00 UTC

The issue was resolved, the disk size should be now 120GiB by default.

Resolved | 2025-09-22 10:47:00 UTC

Resolved, per above comment.

[Resolved] BaseOS CI is unstable

Mon, 08 Sep 2025 02:00:00 +0000

Outage | 2025-09-08 02:00:00 UTC

BaseOS CI is down from early morning. We are looking into the issue.

Update | 2025-09-09 09:00:00 UTC

It was discovered that a Beaker outage is affecting various infrastructure components. Some issues have been resolved, others are still being worked on.

Update | 2025-09-11 08:13:00 UTC

We are experiencing frequent short downtimes of Jenkins. We are trying to resolve the issue.

Update | 2025-09-19 09:40:00 UTC

The issues is still present, we are still working to resolve the issue

Resolved | 2025-09-23 01:44:00 UTC

migration complete, jenkins is up and crunching tasks

[Resolved] Provisioning errors in Public Ranch due to an outage

Mon, 01 Sep 2025 16:00:00 +0000

Outage | 2025-09-01 16:00:00 UTC

The newly created public workers did not have access to Artemis causing provisioning errors during 18:00 UTC to ~ midnight.

Update | 2025-09-02 00:53:00 UTC

The problem was addressed and things should be getting back to normal. Please restart your failed tests.

Resolved | 2025-09-02 17:27:00 UTC

Resolved

[Resolved] Beaker provisioning issues

Mon, 01 Sep 2025 11:50:00 +0000

Outage | 2025-09-01 11:50:00 UTC

There is a wide problem for Beaker causing provisioning issues. Affecting at least s390x. See https://redhat.service-now.com/surl.do?n=INC3762615 for details.

Resolved | 2025-09-03 17:32:00 UTC

Outage was resolved in the evening of 3rd September

[Resolved] Testing Farm public ranch large queue

Mon, 01 Sep 2025 03:00:00 +0000

Outage | 2025-09-01 03:00:00 UTC

We are aware of a large queue. We are investigating the issue. Sorry for the delay :(

Resolved | 2025-09-01 16:34:00 UTC

The outage was resolved by scaling up the deployment. The large queue although is a bit mystery. Anyway, I did not find any direct indication of a problem.

[Resolved] BaseOS CI failing on wow

Thu, 21 Aug 2025 17:00:00 +0000

Outage | 2025-08-21 17:00:00 UTC

Due to the TCMS migration, the old connection URL does not work anymore and BaseOS CI is not able to connect to TCMS via wow. We are working out the issue.

Resolved | 2025-09-01 11:50:00 UTC

BaseOS CI worker image was updated to latest qa-tools, effectively resolving the issue.

[Resolved] Fedora provisioning errors

Tue, 19 Aug 2025 17:23:00 +0000

Outage | 2025-08-19 17:23:00 UTC

We are seeing elevated provisioning issues on Fedora Rawhide mostly. The issues is under investigation.

Update | 2025-08-26 17:53:00 UTC

The issue was found, it is caused by a bug in cloud-init causing wrong ordering of services at startup, breaking in some ocassions sshd. See https://bugzilla.redhat.com/show_bug.cgi?id=2390898 for details. We are waiting for Fedora-Rawhide and Fedora-43 fixed builds to land. Hopefully that will stabilize the testing.

Resolved | 2025-09-01 11:49:00 UTC

This is finally resolved. Both Rawhide and F43 should be stable now.

[Resolved] Fedora CI Jenkins is down

Mon, 18 Aug 2025 08:39:00 +0000

Outage | 2025-08-18 08:39:00 UTC

We are investigating the root cause. No ETA yet.

Update | 2025-08-18 09:10:00 UTC

The Jenkins is back, but it is processing all the requests, can take some time.

Update | 2025-08-18 09:56:00 UTC

We believe Jenkins is processing stuff, but it is overloaded, so expect delays. We will be monitoring the situation and report back here.

Update | 2025-08-18 14:17:00 UTC

Unfortunately the web interface takes too long to load, but the Jenkins is alive. We are trying to scale up the deployment to be able to deal with this load. ETA 1h.

Update | 2025-08-18 17:45:00 UTC

We bumped the memory (8 -> 24Gi) and CPU (3 -> 6) for the instance.

Update | 2025-08-18 18:18:00 UTC

Jenkins is back, UI is responsive, queue is large but going down. We will be monitoring the situation for the next hours. Please be patient. We know that there will be some failed jobs resulting as missing tests in Bodhi and Pagure. Feel free to restart all jobs if you are in hurry.

Resolved | 2025-08-18 20:13:00 UTC

Job queue has finished processing. No obvious issues (apart from the existing ones) were found. If you find anything amiss let us know on the #fedora-ci channel.

[Resolved] Red Hat Gitlab Access Limited

Wed, 13 Aug 2025 09:00:00 +0000

Outage | 2025-08-13 09:00:00 UTC

The Gitlab team is workning to resolve this. You may see HTTP 429 responses in your logs and timeouts. If urgent, please restart the test job.

Resolved | 2025-08-14 07:03:00 UTC

Seems the outage is resolved. Gitlab.cee and pipelines work now well.

[Resolved] baseos-ci queue

Fri, 08 Aug 2025 08:17:00 +0000

Outage | 2025-08-08 08:17:00 UTC

BaseOS-CI have issues with auto-scaling, leading to long queue. It will take at least few hours to process the queue

Resolved | 2025-08-18 08:38:00 UTC

Resolving this old outage. The queue was resolved during the day of outage.

[Resolved] Public Ranch outage

Tue, 05 Aug 2025 20:00:00 +0000

Outage | 2025-08-05 20:00:00 UTC

Due to a broken patch the worker images do not start causing the requests to stay in queued state. We are working out to resolve the issue. It might take some time to process all the requests.

Update | 2025-08-06 09:57:00 UTC

The outage is resolved, we are restarting queued up jobs in batches, can take some time to process. Feel free to restart your testing if you are in hurry. We will resolve the outage once we processed the whole queue.

Update | 2025-08-06 21:00:00 UTC

We restarted all the remaining jobs, expect larger queue in the next hours.

Update | 2025-08-18 08:37:00 UTC

Closing, outage resovled as previously announced.

[Resolved] Fedora Infra outage affecting koji and dist-git

Mon, 04 Aug 2025 06:00:00 +0000

Outage | 2025-08-04 06:00:00 UTC

We are aware of outage of Fedora Infrastructure. The testing will be blocked or delayed. No ETA yet, we are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Resolved | 2025-08-05 08:29:00 UTC

The outage is resolved, Fedora infra should be back.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Thu, 31 Jul 2025 07:00:00 +0000

Outage 2025-07-31 07:00:00 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits.

Resolved 2025-07-31 21:00:00 UTC - The outage s resolved. Please restart your test request if they were affected.

[Resolved] BaseOS-CI stuck pipelines

Thu, 24 Jul 2025 14:45:00 +0000

Outage | 2025-07-24 14:45:00 UTC

last 2 hours we have huge load leading to either provisioning issues or jenkins instability

Update | 2025-07-28 09:09:00 UTC

during weekend, jenkins stopped processing jobs completely. Now it’s restarted and workers are building

Update | 2025-07-28 09:50:00 UTC

jenkins is fully operationl, workers are crunching new jobs. Queue from weekend was corrupted, I will try to reschedule those jobs

Update | 2025-08-02 09:35:00 UTC

most of affected jobs were restarted and processed. Some missed dispatcher to be retriggered

Resolved | 2025-08-18 09:11:00 UTC

Resolved for a long time.

[Resolved] Compose update delayed due to gitlab workers remaining in pending state.

Wed, 23 Jul 2025 09:15:00 +0000

Outage | 2025-07-23 09:15:00 UTC

Intermitent issue. May have minimal impact, but will affect users requiring today’s composes.

Resolved | 2025-07-23 17:26:00 UTC

gitlab workers are running jobs again now with no action from the Testing Farm team.

[Resolved] BaseOS CI and Testing Farm testing blocked on artifacts.osci.redhat.com outage

Wed, 16 Jul 2025 22:00:00 +0000

Outage | 2025-07-16 22:00:00 UTC

The artifacts.osci.redhat.com was migrated to ITUP infrastructure, and the NFS shares did not make it. Basically archiving is impossible, and that is making all the jobs blocked. No ETA currently, folks will be looking into it in the morning.

Update | 2025-07-17 09:25:00 UTC

We are working with ITUP to revive the artifacts storage. No ETA available yet.

Update | 2025-07-17 14:35:00 UTC

We are working with OSCI and ITUP on creating a new NFS volume and migrating a part of artifacts data. No ETA available yet.

Update | 2025-07-17 19:00:00 UTC

The investigation and work with OSCI and ITUP continues. Unfortunately, no ETA on resolving the issue yet :(

Update | 2025-07-18 08:00:00 UTC

The artifact storage has been migrated once again, and the NFS response is back to where it needs to be. Both Red Hat Ranch and Base OS CI were unblocked and started crunching their queue. We are monitoring the situation before the outage gets resolved properly.

Resolved | 2025-07-18 10:30:00 UTC

Resolved. The storage is stable, and the queue is getting smaller.

[Resolved] Public Ranch deployed to 2025-06.1 release

Mon, 14 Jul 2025 21:00:00 +0000

Outage | 2025-07-14 21:00:00 UTC

We deployed a new Testing Farm release. If you see some issues let us know please.

Resolved | 2025-07-17 01:07:00 UTC

Resolved

[Resolved] OpenStack outage

Mon, 14 Jul 2025 08:00:00 +0000

Outage | 2025-07-14 08:00:00 UTC

OpenStack is not operational and affects testing on machines managed by OpenStack.

Resolved | 2025-07-15 21:00:00 UTC

Image listing problem found and resolved.

[Resolved] BaseOS CI artifacts upload not working

Sat, 12 Jul 2025 08:00:00 +0000

Outage | 2025-07-14 07:54:00 UTC

Due to migration of artifacts.osci.redhat.com to ITUP we are not able to upload logs from BaseOS CI to the artifacts storage. This makes some logs unvailable, but otherwise BaseOS CI should be working. We are working out the issue with OSCI, but currently there is no ETA.

Resolved | 2025-07-14 10:10:00 UTC

The outage was resolved via https://redhat.service-now.com/help?id=rh_ticket&table=incident&sys_id=c146c1a8fb366e18ece2f7744eefdc32

[Resolved] Storage almost full - purging

Wed, 09 Jul 2025 07:01:00 +0000

Outage | 2025-07-09 07:01:00 UTC

Disk space is almost full. Old very large artifacts will be purged.

Resolved | 2025-07-09 09:17:00 UTC

Available archive space increased.

[Resolved] BaseOS CI auto-scaling broken, testing stuck

Mon, 30 Jun 2025 09:08:00 +0000

Outage | 2025-06-30 09:08:00 UTC

BaseOS CI testing is currently blocked, because autoscaling does not work, we are looking into the issue.

Resolved | 2025-06-30 10:13:00 UTC

Resolved, the autoscaling still nto working, but we were able to create a worker manually.

[Resolved] src.fedoraproject.org is down

Thu, 26 Jun 2025 10:00:00 +0000

Outage | 2025-06-26 10:00:00 UTC

DC reloctation outage expected. Pleases see status here https://status.fedoraproject.org/ CentOS pkg repos are also affected.

Resolved | 2025-07-14 07:53:00 UTC

The outage is resolved. All Fedora systems are back to normal. Resolving outage.

[Resolved] src.fedoraproject.org down

Tue, 17 Jun 2025 11:11:00 +0000

Outage | 2025-06-17 10:00:00 UTC

src.fedoraproject.org is down and any jobs relying on this resource will fail. Please contact the fedora team about this.

Resolved | 2025-06-17 11:23:00 UTC

Site back online.

[Resolved] BaseOS CI stability issues

Tue, 17 Jun 2025 08:24:00 +0000

Outage | 2025-06-17 08:24:00 UTC

Note that we are fighting with BaseOS CI stability due to the migration to ITUP. If you encounted any issues please contact us on #testing-farm Slack thread.

Update | 2025-06-20 12:52:00 UTC

Seems we are more stable now, keeping the outage open for a few more days.

Resolved | 2025-06-30 09:08:00 UTC

This issue was resolved, BaseOS CI should be more stable now.

[Resolved] Stuck artemis in redhat ranch

Sat, 14 Jun 2025 12:09:00 +0000

Outage | 2025-06-14 12:09:00 UTC

provisioning is stuck and queue is growing

Resolved | 2025-06-14 13:21:00 UTC

unblocked by redeployment, queue should be processed soon

[Resolved] Red Hat Ranch and BaseOS CI not working due to provisioner outage

Fri, 13 Jun 2025 22:00:00 +0000

Outage | 2025-06-13 22:00:00 UTC

Provisioner does nto work, blocking processing all requests on Red Hat Ranch and BaseOS CI

Update | 2025-06-15 11:29:00 UTC

The root cause of the issue was identified and hotfixed. We have a fix so we mitigate this issue to happen again. If you encounter any unexpected provisioning or timeout errors in your test requests, please restart them via your CI system or manually.

Resolved | 2025-06-17 08:23:00 UTC

The outage was resolved on 15th, closing outage.

[Resolved] IT resilience testing multiple SP services down

Fri, 13 Jun 2025 13:00:00 +0000

Outage | 2025-06-13 13:00:00 UTC

tps-server

Resolved | 2025-06-13 21:00:00 UTC

Testing complete

[Resolved] Fedora Rawhide broken due to Python3.14 upgrade

Fri, 13 Jun 2025 10:21:00 +0000

Outage | 2025-06-13 10:21:00 UTC

Unfortunately Fedora Rawhide testing is broken due to Python3.14 which breaks test environment preparation, we are looking how to resolve this with Fedora development, but do not expect it to work today. Sorry for the inconvenience.

Resolved | 2025-06-30 09:14:00 UTC

We believe the testing is now working well in Rawhide. Note that there might be still issue with components broken due to Python3.14 update, but in general TF should be able to provision and run tests against Fedora Rawhide.

[Resolved] Public ranch testing blocked

Thu, 12 Jun 2025 20:00:00 +0000

Outage | 2025-06-12 20:00:00 UTC

The deployment of some workers failed and they cannot access our provisioner Artemis. All jobs provisioning VMs are blocked. We are looking how to reoslve the problem.

Resolved | 2025-06-13 10:20:00 UTC

The outage was resolved.

[Resolved] Testing Farm testing disabled on s390x and ppc64le

Wed, 11 Jun 2025 10:50:00 +0000

Outage | 2025-06-11 10:50:00 UTC

Due to the state of Beaker, see https://redhat.service-now.com/surl.do?n=INC3687996, we have disabled testing on s390x and ppc64le. All requests for these architectures won’t be accepted on the API level.

Resolved | 2025-06-13 09:25:00 UTC

Issues with beaker should be resolved, s390x is not aborting any more.

[Resolved] BaseOS-CI restart

Tue, 10 Jun 2025 20:17:00 +0000

Outage | 2025-06-10 20:17:00 UTC

To update jenkins controller configuration, I need to restart jenkins. Queue suspended.

Resolved | 2025-06-10 20:46:00 UTC

jenkins restarted and working

[Resolved] Disabled direct beaker pipelines in BaseOS-CI

Tue, 10 Jun 2025 12:20:00 +0000

Outage | 2025-06-10 12:20:00 UTC

there is high load in beaker, instability in websocket between jenkins agents and controller and beaker monthly updates. Disabling jenkins jobs for beaker.

Update | 2025-06-10 13:08:00 UTC

canceling all jobs owned by tft-automation-prod

Update | 2025-06-10 15:30:00 UTC

finaly canceled beaker jobs, previous attempt was blocked by beaker maintenance, restarting jenkins with some network related changes

Resolved | 2025-06-13 08:15:00 UTC

this issue seems to be fixed, pipelines enabled again

[Resolved] ppc64le testing blocked

Mon, 09 Jun 2025 07:57:00 +0000

Outage | 2025-06-09 07:57:00 UTC

becaues of long queue time in beaker, ppc64le is blocked

Update | 2025-06-09 09:43:00 UTC

many ppc64le requests running for long time canceled

Resolved | 2025-06-13 08:14:00 UTC

queue is almost empty, ppc64le testing shoud work well. In case of aborted job because of timeout, restart your tests

[Resolved] RHEL modules testing broken

Thu, 05 Jun 2025 21:00:00 +0000

Outage | 2025-06-05 21:00:00 UTC

Changed keytab cause lost of permissions for ODCS

Resolved | 2025-06-06 13:03:00 UTC

ODCS access has been granted and it works from BaseOS-CI again. All affected jobs restarted.

[Resolved] workflow-tomorrow tests will not run

Wed, 04 Jun 2025 08:00:00 +0000

Outage | 2025-06-04 08:00:00 UTC

Due to decommision of REDHAT.COM kerberos realm, workflow-tomorrow tests do not run.

Update | 2025-06-05 17:49:00 UTC

new keytab is deployed and tested, restarting affected jobs. There is ~ 300 jobs, so it will take many hours to process

Update | 2025-06-05 20:11:00 UTC

wow works again, most of affected jobs restarted, pending issue with ODCS access

Resolved | 2025-06-06 13:00:00 UTC

new keytab is working as expected

[Resolved] Beaker outage

Wed, 04 Jun 2025 07:33:00 +0000

Outage | 2025-06-04 07:33:00 UTC

Beaker machines are unavailable due to a connectivity problem. Investigation is ongoing.

Update | 2025-06-04 08:50:00 UTC

Authentication update needed.

Update | 2025-06-04 08:56:00 UTC

Authentication updated… testing.

Update | 2025-06-04 11:35:00 UTC

Confirmed working.

Resolved | 2025-06-04 11:25:00 UTC

Change deployed to production.

[Resolved] RHEL-9.6.0 and RHEL-10.0 image update images are outdated

Tue, 03 Jun 2025 10:00:00 +0000

Outage | 2025-06-03 10:00:00 UTC

The source images from bootc our outdated. Please see https://issues.redhat.com/browse/BIFROST-768 for details. We are blocked until this issue is resolved.

Resolved | 2025-06-30 09:07:00 UTC

The root case was fixed, images are not current. Resolving.

[Resolved] RHEL9 and RHEL10 compose update blocked on IBM cloud outage

Mon, 02 Jun 2025 13:00:00 +0000

Outage | 2025-06-02 13:00:00 UTC

IBM Cloud outage is blocking updating composes for RHEL9 and RHEL10. We are waiting for https://groups.google.com/a/redhat.com/g/outage-list/c/u93AOmuaKqM to be resolved to resume the process.

Update | 2025-06-04 10:20:00 UTC

The issue was resolved yesterday, but it is back again today :(

Resolved | 2025-06-11 08:20:00 UTC

Seems the issue is resolved.

[Resolved] Planned BaseOS-CI outage - migration

Thu, 29 May 2025 19:31:00 +0000

Outage | 2025-05-29 19:31:00 UTC

suspended queue, migration will start when there will be less running jobs

Update | 2025-05-30 01:07:00 UTC

migrated instance up and running, it will take some time to consume queue

Resolved | 2025-05-30 01:44:00 UTC

everything work as expected, queue almost consunemd

[Resolved] Public ranch provisioning is blocked

Mon, 26 May 2025 14:00:00 +0000

Outage | 2025-05-26 14:00:00 UTC

The requests will be stuck in running state until resolved.

Update | 2025-05-26 18:31:00 UTC

The issue was indentified, but we are not able to yet find a workaround. No ETA currently available.

Update | 2025-05-26 19:11:00 UTC

The outage was resolved. Currently we do not see any effects of the outage. If you run into provisioning errors, please restart your requests.

Resolved | 2025-05-27 06:58:00 UTC

Outage was resolved.

[Resolved] Artemis issue with policies

Thu, 22 May 2025 09:07:00 +0000

Outage | 2025-05-22 09:07:00 UTC

Investigating root cause and reverting recent changes. Requests are stuck in routing and will be processed after revert

Update | 2025-05-22 11:52:00 UTC

Issue fixed, queue of guests in routing is going down

Resolved | 2025-05-22 12:57:00 UTC

All requests stuck in routing are now processed. In case of timeout, please restart your test.

[Resolved] Some rpminspect jobs terminated on out-of-memory

Wed, 21 May 2025 06:00:00 +0000

Outage | 2025-05-21 06:00:00 UTC

We are aware of an issue causing rpminspect jobs to be terminted on out-of-memory events even if they have abidiff check disabled. We have rolled back the oom monitoring and are working on a fix in https://issues.redhat.com/browse/TFT-3380. Please restart any jobs if you encounter any falsly terminated jobs. They should be passing now.

Resolved | 2025-05-26 18:24:00 UTC

The issue was addressed and only certain requests should be now termindated and the documentation should be correct. Resolving.

[Resolved] ansible-core breaks ansible-lint

Tue, 20 May 2025 07:42:00 +0000

Outage | 2025-05-20 07:42:00 UTC

Tooling affected by https://github.com/ansible/ansible-lint/issues/4611

Update | 2025-08-07 12:00:00 UTC

This has been mostly mitigated by forcing the ansible version to something before 2.19.

Update | 2025-08-18 09:12:00 UTC

Seems resolved for some time now.

Resolved | 2025-08-19 17:20:00 UTC

Resolving

[Resolved] Instances gone during test execution

Tue, 20 May 2025 06:00:00 +0000

Outage | 2025-05-20 06:00:00 UTC

The spot instances are getting lately terminated more then expected. We are moving to non-spot instances by default, see https://issues.redhat.com/browse/TFT-3371 for details.

Resolved | 2025-05-22 18:01:00 UTC

The issue should be resolved, we are now using non-spot instances by default.

[Resolved] Gitlab.com outage causing various requests to fail

Mon, 19 May 2025 21:00:00 +0000

Outage | 2025-05-19 21:00:00 UTC

Gitlab.com has a wide outage, see https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/682bb3b0158484053205f59b. If the requests use gitlab.com, they might randomly fail on 500 and similar.

Resolved | 2025-05-20 18:00:00 UTC

Things look more stable now. Resolving.

[Resolved] rpminspect requests failing on out-of-memory

Sat, 17 May 2025 18:00:00 +0000

Outage | 2025-05-17 18:00:00 UTC

Due to a deployment error the rollback of a change did not happen, and some rpminspect jobs could be terminated on out-of-memory.

Update | 2025-05-18 23:51:00 UTC

The deployment was fixed, the issue should not appear anymore.

Update | 2025-05-18 23:00:00 UTC

Issue was addressed by rolling back the jobs change.

Resolved | 2025-05-18 23:00:00 UTC

Outage resolved.

[Resolved] BaseOS-CI outage - jenkins migration

Sat, 17 May 2025 11:00:00 +0000

Outage | 2025-05-17 11:00:00 UTC

Stopping jenkins and starting vm migration

Resolved | 2025-05-17 15:23:00 UTC

Normal operation resumed

[Resolved] Testing Farm Red Hat ranch high queue

Thu, 15 May 2025 15:41:00 +0000

Outage | 2025-05-15 15:41:00 UTC

We are experiencing high load on Testing Farm, expect delays on your requests.

Resolved | 2025-05-18 23:48:00 UTC

The queue should be back to normal now.

[Resolved] Beaker outage

Tue, 13 May 2025 14:18:00 +0000

Outage | 2025-05-13 14:18:00 UTC

Beaker outage affects all SUT/DUTs from Beaker.

Resolved | 2025-05-14 07:02:00 UTC

Beaker reboots complete.

[Resolved] Rawhide requests erroring out on Testing Farm

Mon, 12 May 2025 12:00:00 +0000

Outage | 2025-05-12 12:00:00 UTC

We are experiencing high error rate on Public ranch.

Update | 2025-05-12 20:31:00 UTC

Unfortunately the Rawhide composes have expired, we did not notice the update job failing :(

Resolved | 2025-05-12 21:18:00 UTC

The outage was resolved. Please restart any jobs/requests against Rawhide.

[Resolved] BaseOS-CI outage - migration

Sat, 10 May 2025 18:35:00 +0000

Outage | 2025-05-10 18:35:00 UTC

Another try to migrate vm. I expect several hour outage.

Resolved | 2025-05-11 09:00:00 UTC

image transfer completed, starting jenkins

[Resolved] BaseOS-CI outage - migration 2nd try

Fri, 09 May 2025 20:43:00 +0000

Outage | 2025-05-09 20:43:00 UTC

shuting down jenkins controller for vm migration

Resolved | 2025-05-10 09:25:00 UTC

jenkins controller is working again, closing outage

[Resolved] Planned BaseoOS-CI outage (part of migration)

Thu, 08 May 2025 19:00:00 +0000

Outage | 2025-05-08 12:00:00 UTC

Ongoing preparatin to minimize downtime. In case this would cause you issue, let me know (optak)

Update | 2025-05-08 19:00:00 UTC

queue suspended

Update | 2025-05-08 20:21:00 UTC

migration begins

Resolved | 2025-05-08 23:01:00 UTC

outage is over, all affected jobs restarted

[Resolved] BRQ beaker lab aborting jobs

Mon, 05 May 2025 11:00:00 +0000

Outage | 2025-05-05 11:00:00 UTC

jobs ara aborting because of disk space issue

Resolved | 2025-05-05 19:33:00 UTC

resolved by beaker team

NVIDIA V100 GPU not available

Mon, 05 May 2025 06:00:00 +0000

Outage | 2025-05-05 06:00:00 UTC

Unfortunately the instances with NVIDIA V100 GPUs are not available in the availability zone we use. AWS is working on scaling up to meet the demand, but there is currently no ETA. Use the NVIDIA K80 instances if you can, if not please wait, we are working on moving our workloads to multiple AZs, but it might take a few days :( Sorry for the inconvenience.

[Resolved] Red Hat Ranch blocked on high queue

Wed, 23 Apr 2025 22:00:00 +0000

Outage | 2025-04-23 22:00:00 UTC

be patient until we resolve the issue and unblock the queue processing.

Update | 2025-04-24 12:37:00 UTC

The queue is getting better, see http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?orgId=1&refresh=1m&viewPanel=36 for details.

Resolved | 2025-04-25 21:04:00 UTC

Resolving the outage. Testing Farm is stable now. Note that there is still possibility of higher load, as we have users submitting hundreds of jobs at once.

[Resolved] RHEL-9.6.0 not schedulable on Testing Farm

Thu, 17 Apr 2025 15:24:00 +0000

Outage | 2025-04-17 15:24:00 UTC

deployment change. Please start your testing again if you were affected.

Update | 2025-04-22 11:28:00 UTC

Actually the problem affected only Testing Farm.

Update | 2025-04-22 11:30:00 UTC

We asked OSCI to rerun everything from 17th. So RHEL gating should get automatically retried.

Resolved | 2025-04-17 18:00:00 UTC

The compose was unblocked in Testing Farm. Outage resolved.

[Resolved] BaseOS CI break with disk space hardware requirements

Thu, 17 Apr 2025 08:00:00 +0000

Outage | 2025-04-17 08:00:00 UTC

We are aware of an issue with BaseOS-CI causing the jobs with hardware requirements for disk space to fail. We are working to resolve the issue.

Resolved | 2025-04-18 08:00:00 UTC

All affected jobs were restarted. Closing outage.

[Resolved] Some requests cancelled by an error.

Mon, 14 Apr 2025 09:52:00 +0000

Outage | 2025-04-14 09:52:00 UTC

Some of the requests might have canceled by admin error. Please restart your testing if your request was canceled unexpected. We are sorry for the inconvenience.

Resolved | 2025-04-14 10:00:00 UTC

Resolving this notice after 3 days.

[Resolved] Artifacts storage outage for BaseOS CI and Testing Farm

Fri, 11 Apr 2025 13:00:00 +0000

Outage | 2025-04-11 13:00:00 UTC

The artifacts storage was restarted and lost all mount points including Testing Farm and BaseOS CI artifacts storage.

Resolved | 2025-04-11 15:10:00 UTC

The used DNS servers were removed by IT last year. The servers were corrected, mounts points remounted and the outage should be cleared. We are planning to move away from this artifacts storage for better experience.

[Resolved] High load in testing-farm

Wed, 09 Apr 2025 16:00:00 +0000

Outage | 2025-04-09 16:00:00 UTC

High load and queue, expect delays.

Resolved | 2025-04-09 19:26:00 UTC

TestingFarm queue is processed.

[Resolved] Beaker Outage on Red Hat ranch

Tue, 08 Apr 2025 10:00:00 +0000

Outage | 2025-04-08 10:00:00 UTC

Red Hat ranch disruptance due to Beaker pool monthly reboot. ETA - 4 hours.

Update | 2025-04-08 21:57:00 UTC

We are investigating high queue most probably caused by the Beaker outage.

Update | 2025-04-08 23:25:00 UTC

In case you hit some weird errors, make sure to restart.

Update | 2025-04-09 16:26:00 UTC

Queue length is still over 4 hours for ppc64le

Update | 2025-04-10 02:31:00 UTC

Queue length is still over 6 hours for ppc64le. But all jobs on Testing Farm are processed and waiting for free machines in case they landed in Beaker. We are keeping open the outage for awareness.

Resolved | 2025-04-14 09:50:00 UTC

If you see some unexpected results, please restart your testing.

[Resolved] Large queue on Red Hat ranch

Mon, 07 Apr 2025 16:02:00 +0000

Outage | 2025-04-07 16:02:00 UTC

Expect larger queue, we are working on scaling up Testing Farm to deal with the queue.

Resolved | 2025-04-07 20:40:00 UTC

The queue is gone, pipeline should be back to normal.

[Resolved] RHEL-10.0 is expired on Red Hat Ranch

Mon, 07 Apr 2025 07:58:00 +0000

Outage | 2025-04-07 07:58:00 UTC

We are working to resolve the issue.

Update | 2025-04-07 09:42:00 UTC

Images are built, we are working on releasing them. ETA 1h.

Update | 2025-04-07 11:11:00 UTC

Updated compose is deployed to production.

[Resolved] Testing Farm outage due to deployment

Thu, 03 Apr 2025 07:30:00 +0000

Resolved 2025-04-03 9:00 UTC - Testing Farm 2025-01.4 is released. Queue getting crunched

Outage 2025-04-03 7:30 UTC - Both ranches are disabled due to messed up deployment. Expect long queue times.

[Resolved] Red Hat Ranch - expect delays for RHEL-9.5.0 and RHEL-10.0

Thu, 03 Apr 2025 04:00:00 +0000

Notice | 2025-04-03 04:00:00 UTC

Note that due to accidentally removed images for RHEL-10.0 and RHEL-9.5 you can get some unexpected delays. The issue should be addressed and we believe the jobs will start slowly resuming.

Resolved | 2025-04-03 16:00:00 UTC

Images were updated, resolving notice.

[Resolved] Public Ranch Outage - outdated composes

Tue, 01 Apr 2025 14:00:00 +0000

Resolved 2025-04-01 14:47 UTC - The outage is resolved. Outage 2025-04-01 14:00 UTC - Public Ranch provisioning is disrupted due to outdated images, fix is landing in 20 minutes.

[Resolved] Planned brew outage

Fri, 28 Mar 2025 10:00:00 +0000

Outage 2025-03-28 10:00 UTC - Start of planned Brew outage window. BaseOS-CI is suspended, Red Hat Ranch is affected for manually triggered brew task testing

Update 2025-03-28 14:15 UTC - Some complication, unplanned outage right after end of planed one

Resolved 2025-03-29 12:00 UTC - Resuming queue (meanwhile jenkins controller has been upgraded)

[Resolved] Red Hat Ranch Outage - provisioning constrained

Mon, 17 Mar 2025 09:00:00 +0000

Resolved 2025-03-18 08:00 UTC - The problem was addressed, make sure to restart your testing if you were affected by the outage. Sorry for the incovenience.

Outage 2025-03-17 09:00 UTC - Red Hat Ranch testing is disrupted due to provisioning being affected by inablity to talk to services protected by Kerberos. This severly reduces the number of workers taking care of provisioning, and disables Beaker-backed provisioning.

[Resolved] Fedora broken on aarch64, s390x and ppc64le in Beaker

Mon, 10 Mar 2025 06:00:00 +0000

Outage | 2025-03-10 06:00:00 UTC

Fedora testing on s390x, ppc64le and aarch64 is broken. Please share your concern in https://redhat.service-now.com/surl.do?n=INC3438634.

Update | 2025-05-27 08:20:00 UTC

The issue is still present. We have users who are failing to conduct their testing in Fedora-Rawhide and Fedora-42.

Update | 2025-11-03 10:43:00 UTC

This issue is still present. Please update the service now help ticket.

Resolved | 2025-11-27 15:48:00 UTC

The outage is resolved finally. The distros should be installable without issues in Beaker.

[Resolved] Red Hat Ranch Outage - artifacts storage not accessible

Fri, 07 Mar 2025 10:00:00 +0000

Resolved 2025-03-07 16:00 UTC - The DNS records on artifacts.osci.redhat.com were outdated for months. Todays maintanance exposed this problem by causing an outage. Requests will not resume, please restart your requests.

Outage 2025-03-07 10:00 UTC - Red Hat Ranch testing is disrupted due to artifacts storage (artifacts.osci.redhat.com) inacessible. This causes the requests to end up with empty or incomplete artifacts results.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Thu, 06 Mar 2025 15:30:00 +0000

Resolved 2025-03-06 15:30 UTC - This problem is not yet addressed completely. But we are taking care of it manually.

Outage 2025-04-03 7:30 UTC - Both ranches are disabled due to messed up deployment. Expect long queue times.

Update 2025-03-06 22:00 UTC - Provisioning unblocked by deleting some security groups, we will monitor this.

[Resolved] OpenStack outage in Red Hat ranch

Thu, 27 Feb 2025 12:00:00 +0000

Resolved 20205-02-27 18:00 UTC - Everything is back to normal.

Outage 2025-02-27 12:00 UTC - Red Hat ranch is disrupted due to OpenStack outage. Pool is not reachable and artifact pages are not accesible.

[Resolved] RHIVOS HW pool outage - several issues

Mon, 24 Feb 2025 08:28:04 +0000

Outage 2025-02-24 08:00 UTC - ride4-atc and rcar-s4-atc pools are discovered to be problematic.

Update 2025-02-24 11:00 UTC - rcar-s4-atc discovered to have been restarted automatically by systemd on Friday and did restart correctly. Fix is being worked on. ride4-atc pool investigation continues.

Update 2025-02-24 13:00 UTC - ride4-atc and rcar-s4-atc restored. ride4-atc pool also suffered catastrophic restarts. Investigation continues.

[Resolved] Beaker Outage - several issues

Wed, 12 Feb 2025 08:00:00 +0000

Outage 2025-02-12 08:00 UTC - New release of restraint broke reservation

Update 2025-02-17 20:00 UTC - Issue with restraint solved, but beaker is not in a good shape: long queues, many machines marked as broken

Update 2025-02-18 16:00 UTC - Beaker is getting well, but we have some composes outdated, new ones bloced by issue with installing qa-tools

Resolved 2025-02-19 09:00 UTC - All expired composes updated, everything works as expected, beaker queues < 4h

[Resolved] Beaker Outage

Tue, 11 Feb 2025 14:00:00 +0000

Outage 2025-02-11 14:00 UTC - All workloads run on Beaker are affected. The exact time the outage begin is not yet known.

Resolved 2025-02-11 18:00 UTC - Beaker updates/changes/restarts are finally complete.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Fri, 07 Feb 2025 14:00:00 +0000

Resolved 2025-02-07 16:00 UTC - Issue was addressed, but the root cause - a bug in the provisioner is not yet fixed. We hope to fix it in next release.

Outage 2025-02-07 14:00 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits.

[Resolved] Public Ranch Outage - expired composes

Sun, 02 Feb 2025 04:00:00 +0000

Resolved 2025-02-03 08:00 UTC - Images had been refreshed. Please restart your testing, all jobs affected will error out eventually.

Outage 2025-02-02 04:00 UTC - Public ranch composes have expired, we are working on refreshing them. Sorry for the inconvenience.

[Resolved] Red Hat Ranch Outage - issue with Artemis deployment

Fri, 31 Jan 2025 18:00:00 +0000

Resolved 2025-02-03 02:00 UTC - All running jobs timed out, please restart your testing. Thinks should be stable now. We are looking into how to prevent the situation.

Outage 2025-01-31 18:00 UTC - Requests in Red Hat Ranch are stuck due to Artemis outage.

[Resolved] Public Ranch Outage - artifacts storage out of space

Wed, 29 Jan 2025 16:00:00 +0000

Outage 18:00 UTC - Resolve outage, it was resolved in ~ 2 hours.

Outage 16:00 UTC - Artifacts storage on public storage went full. We are working on the issue.

[Resolved] Red Hat Ranch Outage - issue with Artemis deployment

Fri, 17 Jan 2025 09:00:00 +0000

Outage 2025-01-17 09:00 UTC - Requests in Red Hat Ranch are stuck due to Artemis outage.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Thu, 16 Jan 2025 15:20:00 +0000

Resolved 2025-01-16 16:10 UTC - The outage s resolved. Please restart your test request if they were affected.

Outage 2025-01-16 15:20 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits. ETA is 30 minutes.

[Resolved] Red Hat Ranch Outage - issue with Artemis deployment

Thu, 16 Jan 2025 00:20:00 +0000

Resolved 2025-01-16 04:20 UTC - Artemis deployment recovered and working properly again

Outage 2025-01-16 00:20 UTC - Red Hat ranch blocked by artemis deployment issue ind OpenShift

[Resolved] Fedora Rawhide outdated image

Tue, 14 Jan 2025 08:00:00 +0000

Resolved 2025-01-15 10:00 UTC - New configuration deployed, fixing Rawhide provisioning in public ranch.

Outage 2025-01-14 08:00 UTC - Rawhide image has been deleted and automation for updating looks broken, investigating.

[Resolved] Fedora Rawhide outdated image

Thu, 09 Jan 2025 08:00:00 +0000

Outage 2025-01-09 12:00 UTC - Image updated, outage is resolved.

Outage 2025-01-09 08:00 UTC - Rawhide image has been deleted and new one updated yesterday is alredy missing too, investigating.

[Resolved] Fedora CI is not triggering on events due to expired Fedora message bus

Wed, 18 Dec 2024 09:00:00 +0000

Resolved 2024-12-18 15:00:00 UTC - The certificate was rotated.

Outage 2024-12-18 09:00:00 UTC - Fedora CI is not triggering on events, we are trying to resolve the outage. The certificate for message bus has expired today.

[Resolved] Brew outage https://brewweb.engineering.redhat.com/brew/

Tue, 10 Dec 2024 09:00:00 +0000

Outage 2024-12-19 11:30:00 UTC - Brew service is down. All testing relying on Brew are affected. See https://brewweb.engineering.redhat.com/brew/

[Resolved] Testing Farm fails to authenticate to OpenStack

Tue, 10 Dec 2024 08:00:00 +0000

Outage 2024-12-10 15:00:00 UTC - Resolved by using new credentials for Openstack.

Outage 2024-12-10 08:00:00 UTC - Testing Farm fails to authenticate to OpenStack, we are investigating the root cause.

[Resolved] Resilience Testing outage

Fri, 06 Dec 2024 12:30:00 +0000

Resolved 2024-12-06 19:00:00 UTC - The terting is finished.

Outage 2024-12-06 12:30:00 UTC - Due to Resilience Testinga lot of services, such as brew, mbs, dit-git are down. The planned resolvance time is 2024-12-06 19:00:00

[Resolved] artifacts.osci.redhat.com outage

Thu, 05 Dec 2024 13:00:00 +0000

Resolved 2024-12-05 15:30:00 UTC - The outage was resolved. The issue was that IT falsly shut down the instance as unclaimed. The running jobs could be affected and never end (end up in running state). Please restart your testing if needed. New jobs are not affected from now.

Outage 2024-12-05 13:00:00 UTC - Archiving of artifacts in Red Hat ranch is affected due to outage artifacts.osci.redhat.com.

[Resolved] RHEL-9.6.0-image-mode is broken

Tue, 26 Nov 2024 00:00:00 +0000

Resolved 2024-12-05 15:30 UTC - RHEL-9.6.0-image mode is available on Testing Farm and tmt again. Virtualization support and aarch64 is still in works, but outside of that it should be ready for consumption.

Outage 2024-11-26 00:00 UTC - The image mode MVP testing on RHEL-9.6.0 is currently blocked, the image has expired, we are working this out.

[Resolved] download.devel SSL certificate expired

Sun, 24 Nov 2024 21:00:00 +0000

Resolved 2024-11-25 10:30 UTC - Outage was resolved by IT. Certificate was refreshed.

Outage 2024-11-24 23:00 UTC - See https://redhat.service-now.com/surl.do?n=INC3256074 for details.

Outage 2024-11-24 21:00 UTC - The download.devel SSL certificate has expired, this is affecting BaseOS CI and Testing Farm Red Hat ranch.

[Resolved] changed script for postinstall broke jobs in beaker

Tue, 19 Nov 2024 17:00:00 +0000

Resolved 2024-11-21 07:30 UTC - revert deployed, restart affected jobs if needed

Outage 2024-11-21 07:00 UTC - recent change broke jobs running in beaker

[Resolved] RHEL 10 CTC outage on Red Hat Ranch - broken compose

Tue, 19 Nov 2024 00:00:00 +0000

Resoled 2024-12-04 02:00 UTC - few tooling fixes applied and compose added also as RHEL-10.0-ctc and RHEL-10.0-ctc1

Updata 2024-11-21 00:00 UTC - we already have new compose without this issue: RHEL-10.0-20241120.1, not yet promoted to CTC

Outage 2024-11-19 00:00 UTC - RHEL 10 CTC testing affected on Red Hat Ranch due to broken compose (https://issues.redhat.com/browse/RHEL-66937).

[Resolved] Fedora 41 and Rawhide outage on public ranch - 0xe2 breaking ansible

Fri, 15 Nov 2024 00:00:00 +0000

Resolved 2024-11-18 23:15 UTC - Fix for the issue had been deployed and Fedora-41 and Rawhide should work again.

Update 2024-11-15 13:00 UTC - dnf bug triggering our bug: https://github.com/rpm-software-management/dnf5/issues/1865

Update 2024-11-15 11:20 UTC - fix and test is ready, working on deployment

Update 2024-11-15 11:00 UTC - issue is in ensure_str(), inside gluetool. Fixing now and working on extending test

Outage 2024-11-15 00:00 UTC - Fedora 41 and rawhide affected by ansible crashed on decoding UTF-8 char 0xe2

[Resolved] Red Hat Ranch / BaseOS CI Outage - provisionig issues during the weekend

Sat, 09 Nov 2024 00:00:00 +0000

Resolved 2024-11-11 10:00 UTC - Queues are consumed, outage is over.

Outage 2024-11-11 09:00 UTC - We are working on resolving the issue in about an hour.

Outage 2024-11-10 00:00 UTC - Requests are blocked and the queue is rising. All requests will timeout out and restarting will be needed after the weekend.

Outage 2024-11-09 00:00 UTC - Due a bug causing the Artemis worker tasks to wait forever to extend reservations in PEK labs, our provisioner went down.

[Resolved] Misconfigured worker in public ranch

Fri, 08 Nov 2024 08:00:00 +0000

Resolved 2024-11-08 15:00 UTC - Outage was resolved, out workers lost connection to Artemis, manual cleanup of SG resolved the issue.

Outage 2024-11-08 08:00 UTC - One of the public workers was misconfigured, causing connection problems for a small number of test number of test requests.

[Resolved] Testing Farm 2024-10.1 released

Thu, 07 Nov 2024 08:00:00 +0000

Resolved 2024-11-11 08:00 UTC - Everything looks ok, end of outage

Update 2024-11-11 09:00 UTC - Release should be stable, the other outages are not related to the release itself. Anyway, keeping this here as a note for one more week to raise awareness in case somebody runs into issues.

Update 2024-11-07 18:30 UTC - We should be stable now and deployed to 2024-10.1. There can be an issue where the results viewer is not loading some older results, no data gone, just it does nto present the data and you get an empty screen, We are trying to resolve it.

Update 2024-11-07 09:30 UTC - A bit of queue can happen. Please be patient.

Update 2024-11-07 09:00 UTC - Note that the new release is introducing some breaking changes, see a preview in the https://gitlab.com/testing-farm/docs/root/-/merge_requests/153[release notes draft].

Update 2024-11-07 08:10 UTC - A short outage caused some jobs to error out on Public Ranch, please restart your jobs.

Info 2024-11-07 08:00 UTC - We are deploying Testing Farm 2024-10.1. We are trying to avoid outages, but if you see something, check this status.

[Resolved] Testing Farm API down

Wed, 06 Nov 2024 14:30:00 +0000

Resolved 2024-11-06 15:10 UTC - The outage is over. API is back. Please restart your jobs if you were affected. We are sorry for the inconvenience.

Outage 2024-11-06 14:30 UTC - We are suffering from an API outage, jobs will break, can be stuck forever.

[Resolved] Fedora 41 outage on Red Hat ranch

Mon, 04 Nov 2024 18:00:00 +0000

Resolved 2024-11-04 13:00 UTC - The issue is resolved, enjoy!

Update 2024-11-04 13:00 UTC - The issue was identified, seems dnf5 broke the environment, and we are working on restoring the support. ETA the next day.

Outage 2024-11-02 13:00 UTC - Fedora 41 does not provision on Red Hat ranch. We are investigating the issue.

[Resolved] Gitlab CEE outage affecting jobs run on BaseOS CI and Red Hat ranch.

Mon, 04 Nov 2024 18:00:00 +0000

Resolved 2024-11-05 03:00 UTC - The outage is over. See outage list for more details.

Outage 2024-11-04 18:00 UTC - GitLab CEE outage affecting a lot of jobs.

[Resolved] BaseOS CI and Red Hat Ranch affected by provisioning outages. BaseOS CI down.

Fri, 01 Nov 2024 00:00:00 +0000

Resolved 2024-11-01 09:00 UTC - Outage was resolved. Please restart jobs if ended in error state.

Outage 2024-11-01 00:00 UTC - BaseOS Jenkins is down, and Red Hat ranch provisioning ending in error state.

[Resolved] Fedora 41 outage on public ranch.

Fri, 01 Nov 2024 00:00:00 +0000

Resolved 2024-11-01 15:00 UTC - Fedora 41 images are updated and working now.

Outage 2024-11-01 00:00 UTC - Fedora 41 images are gone due to a rename, we are working on resolving the issue.

[Resolved] BaseOS CI and Red Hat Ranch affected by OpenShift outage

Thu, 31 Oct 2024 12:00:00 +0000

Partial outage 2024-10-31 12:00 UTC - BaseOS CI and Red Hat Ranch affected by maintenance of RDU2 OpenShift.

[Resolved] BaseOS CI suspended for changes in UMB certificate.

Tue, 15 Oct 2024 17:00:00 +0000

**Resolved 2024-10-15 23:55 UTC - queue resumed, still running jobs canceled & restarted

Outage 2024-10-15 17:00 UTC - queue suspended, except for dispatchers

Planned maintenance 2024-10-15 14:00 UTC - Planned maintenance of jenkins controllers. Impact: suspended queue since 17:00 UTC, expected end: 20:00 UTC

[Resolved] Public Ranch Outage - no archive disk space

Tue, 01 Oct 2024 10:00:00 +0000

Resolved 11:00 UTC - Alert created for archive disk space. Purged old files.

Outage 10:00 UTC - Public Artemis stopped working, investigating. Until this is resolved, request will stay queued and will be processed later.

[Resolved] Public Ranch - broken Fedora repositories

Mon, 16 Sep 2024 00:00:00 +0000

Resolved 2024-09-16 19:00 UTC - The outage was resolved by the Fedora infrastructure team.

Outage 2024-09-16 00:00 UTC - We are experiencing outages due to broken repositories in Fedora. The issue is tracked by Fedora Infrastructure in this issue - https://pagure.io/fedora-infrastructure/issue/12183. The issue causes following failures in Testing Farm:

Updating and loading repositories:                                                            
 Fedora 41 openh264 (From Cisco) - x86_64             100% |   3.0 KiB/s |   6.0 KiB |  00m02s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 385.2 KiB | -00m00s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 385.2 KiB | -00m00s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 385.2 KiB | -00m00s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 391.1 KiB | -00m00s
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 391.1 KiB | -00m00s
...snip...
>>> Downloading successful, but checksum doesn't match. Calculated: f6dc87bb2fab2e324c8c8f3980
>>> Downloading successful, but checksum doesn't match. Calculated: f6dc87bb2fab2e324c8c8f3980
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirror
Failed to download metadata (metalink: "https://mirrors.fedoraproject.org/metalink?repo=fedora-41&arch=x86_64") for repository "fedora"
 Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[Resolved] Red Hat Ranch / BaseOS CI Outage - provisionig issues

Tue, 03 Sep 2024 20:00:00 +0000

Outage 2024-09-04 02:00 UTC - We were not able to resolve the issues during night, your requests might error out or get canceled.

Outage 2024-09-03 20:00 UTC - We are experiencing provisioning issues on Red Hat ranch.

[Resolved] Red Hat Ranch Outage - large logs consuming all the space. Testing-Farm dispatcher is disabled

Fri, 30 Aug 2024 08:30:00 +0000

Outage 2024-08-30 13:30 UTC - issue was identified, outage is resolved, please retru your requests if you they failed unexpectedly or did not finish

Outage 2024-08-30 08:30 UTC - dispatching new jobs is disabled, investigating extensive logging which caused full disk

[Resolved] Public ranch outage - workers not able to connect to Artemis.

Wed, 28 Aug 2024 10:29:00 +0000

Resolved 2024-08-28 14:21 UTC - the service is fully operational. If your requests are still failing, please restart them.

Outage 2024-08-28 10:29 UTC - because of misconfigured network in production environment, workers are not able to connect to Artemis.

[Resolved] Fedora 40 small disk size

Tue, 20 Aug 2024 18:00:00 +0000

Resolved 2024-08-29 08:00:00 UTC - Resolved by updating to new, not broken F40 image

Degradation 2024-08-20 18:00 UTC - Due to recent btrfs changes in Fedora 40, the auto-resizing of root partition to the cloud image is not working. For details see https://bodhi.fedoraproject.org/updates/FEDORA-2024-67f6df918c and https://bodhi.fedoraproject.org/updates/FEDORA-2024-d44bd4abd9 bodhi updates. The issue should be fixed in 2024-08-22 once these fixes get into the fresh compose.

[Resolved] Red Hat Ranch and BaseOS-CI Outage - dns misconfiguration, some crucial sites not accessible

Tue, 13 Aug 2024 15:40:00 +0000

Resolved 2024-08-14 00:50 UTC - both services fully operational.

Update 2024-08-13 23:42 UTC - dns issues looks resolved, baseosci resumed (long queque), testing farm under testing

Outage 2024-08-13 15:50 UTC - because of issue with dns servers, some internal sites are not accessible, BaseOS-CI is suspended to lower impact

[Resolved] Testing Farm API outage and longer queues on both ranches.

Thu, 08 Aug 2024 04:00:00 +0000

Outage 2024-08-08 06:15 UTC - We resumed normal operation. Outage is over.

Outage 2024-08-08 04:00 UTC - We experienced an api outage. If you encounted some issues, please restart your requests.

[Resolved] All Fedora images outdated, public ranch requests processing paused.

Mon, 05 Aug 2024 04:00:00 +0000

Resolved 2024-08-05 13:00 UTC - The outage was resolved by uploading images to Fedora AWS ourselves. We are still waiting for a proper fix on fedora infrasturcture side.

Update 2024-08-05 12:30 UTC - The upload is progressing, we had to implement few features to make the uploaded images look same as with the original ones, e.e. bios mode unset, gp3 used. ETA 1h.

Update 2024-08-05 09:30 UTC - The uploading of the images is broken, no new updates for long time :(. See https://pagure.io/fedora-infrastructure/issue/12110 for details. We stopped processing public ranch queue until the outage is resolved, we are uploading the images manually.

Outage 2024-08-05 08:00 UTC - All Fedora images have expired, we are investigating the issue.

[Resolved] Fedora Rawhide outdated image

Fri, 02 Aug 2024 08:00:00 +0000

Update 2024-08-02 09:00 UTC - New configuration deployed, fixing Rawhide provisioning in public ranch

Outage 2024-08-02 08:00 UTC - Rawhide image has been deleted and automation for updating looks broken, investigating.

[Resolved] Beaker PPC64 Outage - large queue because of limited hardware

Fri, 26 Jul 2024 00:00:00 +0000

Resolved 2024-08-02 00:00 UTC - ppc64le queue times stayed below 4 hours for 3 days.

Outage 2024-07-26 00:00 UTC - due to a deteriorating PPC64 pools in Beaker, testing on this ARCH is not advised. Expect over 24 hours delay.

[Resolved] Public and Redhat Ranch Outage - broken deployment

Thu, 25 Jul 2024 13:00:00 +0000

Update 19:45 UTC - almost all jobs from queue has been processed.

Update 18:18 UTC - deployemnt fixed, queues are being processed but it will take some time.

Outage 13:00 UTC - Issue with deployment affecting all new jobs.

[Resolved] Public Ranch Outage - artifacts out of space

Tue, 23 Jul 2024 06:00:00 +0000

Resolved 10:00 UTC - The issue was resolved. Resolving outage.

Update 09:00 UTC - We started to investigate the issue and working on getting back TF public ranch into a good state.

Outage 06:00 UTC - Testing Farm Artifacts are out of space.

[Resolved] Public Ranch Outage - complete outage, request will stay queued or error out

Mon, 22 Jul 2024 00:16:00 +0000

Resolved 09:17 UTC - The issue was resolved, our postgres database went out of space in the early morning. All affected requests restarted. Fedora CI users of Testing Farm, will need to manually restart their testing. Packit users pipelines should automatically resolve.

Outage 07:17 UTC - Public Artemis stopped working, investigating. Until this is resolved, request will stay queued and will be processed later.

[Resolved] Cyborg Loki is down

Tue, 16 Jul 2024 00:00:00 +0000

Resolved 9:30 UTC - Loki is back up. Unfortunately due to deployment having delete of pvc on scale down, we lost all the data. This misconfiguration was unknown and should not happen anymore.

Outage 0:00 UTC - Loki is down, we are investigating the issues.

[Resolved] Public Ranch - Large queue

Mon, 17 Jun 2024 13:00:00 +0000

Warning 2024-06-17 13:00 UTC - Public ranch has 1k+ queued jobs, seems Fedora CI load, be patient, it is getting progressed, but expect delays.

[Resolved] Red Hat Ranch and BaseOS-CI Outage - broken krb infra

Tue, 28 May 2024 18:00:00 +0000

Resoled 2024-05-29 10:00 UTC - Change related to krb infra reverted, everything works now.

Update 2024-05-29 9:00 UTC - BaseOS-CI partially unblocked by deploying workaround (missing fix in artemis)

Update 2024-05-29 10:00 UTC - something has changed in kerberos infra leading to broken keytab. All requests to beaker are affected

[Resolved] Red Hat Ranch Outage - large queue, jobs getting delayed

Mon, 27 May 2024 00:00:00 +0000

Resolved 2024-05-27 12:00 UTC - the problem had been resolved, resolving

Update 2024-05-27 00:00 UTC - we are investigating the problem, no eta yet, problem started most probably earlier

[Resolved] Red Hat Ranch Outage - large queue, jobs getting delayed

Sun, 19 May 2024 06:00:00 +0000

Resolved 2024-05-20 20:00 UTC - things should be stable, plus rpmdeplint and rpminspect should work again

Update 2024-05-20 11:00 UTC - we got into some weird problem with provisioner, causing workers to get stuck, seems we were able to get provisioner back into a working state. We will soon start enabling back the workers.

Update 2024-05-20 08:00 UTC - we are investigating the problem, no eta yet

Outage 2024-05-19 06:00 UTC - large queue started to appear during the weekend

[Resolved] Public Ranch Outage - possible large queue due to deadlocked workers

Wed, 15 May 2024 20:00:00 +0000

Resolved 2024-05-16 19:30 UTC - problems should be gone

Monitoring 2024-05-16 07:30 UTC - the queue should be gone, we are investigating the root cause

Outage 2024-05-16 01:00 UTC - we started to investigate the issue

Outage 2024-05-15 20:00 UTC - large queue started to appear

[Resolved] Public Ranch Outage - artifacts storage out of space

Thu, 09 May 2024 16:00:00 +0000

Update 19:30 UTC - Old requests with large space usage were removed. Outage resolved.

Outage 16:00 UTC - Artifacts storage on public storage went full.

[Resolved] Public Ranch Outage - F40 images expired

Mon, 06 May 2024 00:00:00 +0000

Update 7:30 UTC - Issue was addressed. F40 image was updated.

Outage 00:00 UTC - F40 image got expired.

[Resolved] Public Ranch Outage - large queue, high error rate due to broken workers

Tue, 30 Apr 2024 18:00:00 +0000

Update 21:30 UTC - Outage was resolved.

Update 20:30 UTC - Issue was addressed. Queue is being processed. Some jobs might have failed, please restart your jobs if they errored out.

Outage 18:00 UTC - We broke our workers and they are not able to connect to our provisioner. We are investigating the issue.

[Resolved] Public Ranch Outage - Broken Fedora Rawhide images

Thu, 25 Apr 2024 00:00:00 +0000

Resolved 11:00 UTC Workaround with installing python3-dnf deployed

Outage 00:00 UTC - Our images in public ranch have broken dnf installation, missing python3-dnf

[Resolved] Public Ranch Outage - Random issues with access to koji

Thu, 25 Apr 2024 00:00:00 +0000

Outage 2024-05-06 7:15 UTC - The issue looks resolved, we did not see it for some time.

Outage 2024-04-25 00:00 UTC - Random connection issues (timeout) for kojihub.stream.centos.org affecting CentOS-Stream images

[Resolved] Public Ranch Outage - Expired images

Mon, 15 Apr 2024 00:00:00 +0000

Resolved 08:30 UTC - Resolved. Images have been updated.

Outage 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Public Ranch Outage - Fedora 40 expired

Fri, 29 Mar 2024 00:00:00 +0000

Resolved 17:00 UTC - Our F40 image has expired.

Outage 00:00 UTC - Our F40 image has expired.

[Resolved] Public Ranch Outage - Expired images

Thu, 28 Mar 2024 00:00:00 +0000

Resolved 09:30 UTC - The images were bumped, sorry for the outage.

Outage 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Red Hat Ranch Degraded

Thu, 28 Mar 2024 00:00:00 +0000

Resolved 22:30 UTC - Beaker is back. Degradation should be over.

Degraded 00:00 UTC - Red Hat Ranch is degraded due to Beaker planned outage (migration to AWS). Jobs for ppc64le, s390x and specific HW requirements satisfied only by Beaker will fail.

[Resolved] Public Ranch Outage - Artemis requests in error state

Tue, 19 Mar 2024 00:00:00 +0000

Outage 11:00 UTC - Images were updated. We are adding finally automation to get them updated regularly.

Outage 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Long queues in Public and Red Hat ranch

Tue, 12 Mar 2024 15:00:00 +0000

Resolved 20:00 UTC (next day) - The problems should be gone. If you see weird failures please restart, unfortunately we had outage with artifacts storage, so rsync might have timed out. Should be all ok now.

Outage 15:00 UTC - Long queues detected, due to workaround gone bad. We are scaling and hopefully the issues should be over. Please restart if you run into some issues. Testing should be otherwise stable.

[Resolved] Public Ranch artifact storage ran out of disk space

Fri, 08 Mar 2024 08:00:00 +0000

Outage 8:00 UTC - Testing Farm public ranch artifact storage ran out of disk space. Some requests might end unexpectedly, feel free to restart them. We are working on cleaning up old large artifacts and increasing disk space.

[Resolved] Public Ranch large queue - expect delays

Thu, 07 Mar 2024 10:00:00 +0000

Resolved* 15:00 UTC - Testing Farm queue is large again due to an outage. Expect delays, ETA 1 hour.

Resolved* 12:00 UTC - Testing Farm is deployed. Queue is processed. Normal operation expected.

Outage 10:00 UTC - Testing Farm public ranch has long queue due to ongoing deployment. It might take a bit of time the queue to go down. Please be patient.

[Resolved] Public Ranch large queue - expect delays

Tue, 05 Mar 2024 15:00:00 +0000

Outage 21:00 UTC - Outage was resolved.

Outage 15:00 UTC - Testing Farm public ranch has long queue due to deployment. It might take a bit of time the queue to go down. Please be patient.

[Resolved] Public Ranch large queue - expect delays

Thu, 15 Feb 2024 17:00:00 +0000

Resolved 19:30 UTC - Queue is almost gone, resolving.

Outage 17:00 UTC - Testing Farm public ranch has long queue due to deployment. It might take a bit of time the queue to go down. Please be patient.

[Resolved] BaseOS CI suspended for urgent jenkins controller upgrade.

Fri, 02 Feb 2024 08:00:00 +0000

Resolved 09:00 UTC - Jenkins controller is running, all affected jobs were restarted.

Outage 08:00 UTC - All jobs (except for dispatchers) suspended as a preparation for jenkins controller upgrade. Expect delays for test result.

[Resolved] BaseOS CI suspended for urgent jenkins controller upgrade.

Wed, 03 Jan 2024 10:47:00 +0000

Resolved 12:30 UTC - Jenkins controller is running, all affected jobs were restarted.

Outage 10:47 UTC - All jobs (except for dispatchers) suspended as a preparation for jenkins controller upgrade. Expect delays for test result.

[Resolved] Public Ranch Outage - Artemis requests in error state

Sun, 10 Dec 2023 00:00:00 +0000

Resolved 2023-12-11 10:00 UTC - The images were updated. The issue should be resolved.

Outage 2023-12-10 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Public Ranch Outage

Sat, 02 Dec 2023 00:00:00 +0000

Outage 2023-12-02 03:00 UTC - Outage was resolved. Please restart your tests if your tests failed unexpectedly.

Outage 2023-12-02 00:00 UTC - We need to reprovision our Kubernetes cluster and we hit some issues trying to do that without outage. Public Ranch will be paused for 12 hours at least. Sorry for the inconvenience.

[Resolved] Public Ranch reservations are broken

Wed, 29 Nov 2023 00:00:00 +0000

Resolved 2023-12-07 10:00 UTC - Deployment has been changed to terragrunt, the issues is resolved, reservations should work according to the IP allowlist.

Info 2023-11-29 00:00 UTC - Due to deployment changes we broke the Testing Farm reservations in public, please bare with us until we resolve the issues. We hope to resolve the problems today.

[Resolved] Fedora CI installability test affected by systemd/selinux issue.

Thu, 23 Nov 2023 00:00:00 +0000

Resolved 2024-02-09 10:00 UTC - The issue is resolved, please let us know if you would have still issues.

Update 2023-11-28 19:00 UTC - The problem still persists. Please follow this bugzzila https://bugzilla.redhat.com/show_bug.cgi?id=2250930 for more details.

Outage 2023-11-23 00:00 UTC - A systemd/selinux issue is causing unexpected failures in installability for a lot of components. Also some of the functional tests might be affected. For more info see https://pagure.io/fedora-ci/general/issue/447

[Resolved] Testing Farm Public Ranch outage due to expired images.

Mon, 20 Nov 2023 00:00:00 +0000

Resolved 10:00 UTC - Outage was resolved, images were updated.

Outage 00:00 UTC - Some of our public images have expired, we are working to get them updated.

[Resolved] Testing Farm Red Hat Ranch outage during the weekend

Sun, 19 Nov 2023 06:00:00 +0000

Resolved 2023-11-20 09:00 UTC - We do not know the root cause, but we will try to improve the liveness probes to autoheal the deployment if the problem happens. Tracked under https://issues.redhat.com/browse/TFT-2353[TFT-2353].

Update 2023-11-20 06:00 UTC - The provisioner is working again, we are trying to find out what happened and how to deal with the problem automatically.

Outage 2023-11-19 06:00 UTC - Our provisioner has stopped processing new requests during the weekend. We are investigating the issue.

[Resolved] BaseOS CI limited due to outage due to BRQ network maintanance.

Sun, 12 Nov 2023 00:00:00 +0000

Resolved 12:00 UTC - Registry is back, outage is resolved. The tests should automatically resume, no manual intervention should be needed.

Outage 00:00 UTC - Our BRQ image registry went down after BRQ network maintanance. This is blocking BaseOS CI autoscaling, so the queue can get large. We are working to resolve the issue.

[Resolved] BaseOS CI and Testing Farm Red Hat Ranch outage due to Beaker unavailability

Tue, 07 Nov 2023 15:00:00 +0000

Resolved 19:00 UTC - Beaker is back, we resumed BaseOS CI and Testing Farm, jobs are processing.

Outage 15:00 UTC - Beaker is down including required repositories which break all executions on RH ranch for RHEL. We disabled the ranch completely to overcome too many failures. Will be posting updates here.

[Resolved] Expired images on Public ranch.

Sat, 30 Sep 2023 04:00:00 +0000

Resolved 2023-10-02 04:00 UTC - Images were updated, problems should be gone. Restart of testing will be needed for affected users.

Outage 2023-09-30 4:00 UTC - Some of the images have expired on Public Ranch.

[Resolved] Testing Farm Red Hat ranch and BaseOS CI outage for RHEL9 and RHEL-8.1.0

Mon, 25 Sep 2023 00:00:00 +0000

Resolved 16:30 UTC - All affected composes updated, the problems should be gone. We will try to auto-restart fail jobs, please feel free to restart yourself also if in hurry.

Update 12:30 UTC - All problems affecting building had been fixed. We are getting fresh composes through. ETA 2-3 hours.

Outage 0:00 UTC - Composes have expired due to a sad misunderstanding from Friday :( We are trying to resolve the issue, but it will take few hours.

[Resolved] BaseOS CI outage due to network issues

Thu, 21 Sep 2023 00:00:00 +0000

Resolved 7:00 UTC - Problem was resolved by IT. BaseOS CI is back in action.

Outage 0:00 UTC - After restart DHCP does not resolve address of the machine. Working with IT to investigate the problem.

[Resolved] Console logs are disabled due to a discovered bug

Mon, 18 Sep 2023 14:00:00 +0000

Resolvede 2023-09-19 21:00 UTC

Issue was resolved in Artemis: https://gitlab.com/testing-farm/artemis/-/merge_requests/1160

Outage 2023-09-18 8:00 UTC

We have found a bug we need to address and had to temporarily disable the console log functionality. ETA 1-2 days.

[Resolved] Red Hat Ranch outage during weekend

Sun, 17 Sep 2023 08:00:00 +0000

Outage 16:00 UTC - Resolved.

Outage 8:00 UTC - We had an outage during the weekend in our provisioner. Some jobs might have failed. Please restart your testing if needed.

[Resolved] Console log has wrong link in results viewer or not available at all.

Fri, 15 Sep 2023 08:00:00 +0000

Resolved 2023-09-26 18:00 UTC

Console log links are now correct with a hotfix. Resolving outage.

Outage 2023-09-15 8:00 UTC

Red Hat ranch: Console log has wrong link in the results viewer for Red Hat ranch. We are looking into the problem. Until then, please remove the work-.* path from the URL. For example http://artifacts.osci.redhat.com/testing-farm/10de2a43-a2a1-4c4f-ab57-cf5e4416c93b/work-create-osbuild1e5bpd1y/console-94d0f6af-27a9-416c-9412-32bcf440dcce.log should be http://artifacts.osci.redhat.com/testing-farm/10de2a43-a2a1-4c4f-ab 57-cf5e4416c93b/console-94d0f6af-27a9-416c-9412-32bcf440dcce.log, note the work-create-osbuild1e5bpd1y path removed in this case to get a valid link.

Public ranch: Console logs are not yet working on Public ranch, we are working to get them available also.

[Resolved] Composes not updating on Red Hat ranch

Mon, 11 Sep 2023 08:00:00 +0000

Update 2023-09-25 12:00 UTC - Composes are being updated now.

Update 2023-09-18 8:00 UTC - Seems we finally have a a way forward here. We hope to fix the issue today finally.

Outage 2023-09-11 8:00 UTC - Composes are not updating properly due to issues with the build process. We are trying to fix the issue.

[Resolved] Cyborg Loki is being migrated to S3 storage and better deployment

Wed, 06 Sep 2023 08:00:00 +0000

Resolved - Resolved by loosing all data and redeployment.

Outage 8:00 UTC - Cyborg Loki is unavailable, it is being migrated to S3 storage and better deployment. ETA 24 - 48 hours

[Resolved] Beaker testing blocked on Red Hat Ranch and BaseOS CI

Mon, 21 Aug 2023 08:00:00 +0000

Resolved 12:00 UTC - The issue was resolved by Beaker admins. Outage is resolved.

Update 9:00 UTC - Seems the issue is only with /distribution/reservesys, the task was disabled in Beaker :) We are waiting for admins to resolve.

Outage 8:00 UTC - All testing against Beaker is blocked due to missing /distribution tests from Beaker. Investigation is ongoing.

[Resolved] Large queue times on Red Hat Ranch

Mon, 07 Aug 2023 00:00:00 +0000

Update 7:22 UTC - TF queue has been consumed, operating as usual.

Investigating 7:22 UTC - We’re aware of an high number of requests affecting queue times on Red Hat Ranch of Testing Farm. Testing can be delayed several hours.

[Resolved] Red Hat Ranch affected by large queue

Thu, 27 Jul 2023 09:00:00 +0000

Resolved - Queue reduced from > 600 to 15

Outage - Adding additional 8 workers.

Outage - ETA 1hr

Outage - Queue is reducing

Outage - Added additional 8 workers.

Outage - Testing jobs may time out due to high load.

Outage - Added additional 8 workers.

Outage - Testing jobs have timed out due to high load.

[Resolved] Public ranch testing blocked

Wed, 19 Jul 2023 08:00:00 +0000

Resolved | 2023-07-19 11:00:00 UTC

Problem was resolved by applying a hotfix to guest-setup playbooks.

Outage | 2023-07-19 08:00:00 UTC

We are trying to update the images for Rawhide, but it breaks due to dnf5 again. Testing is completely blocked until we resolve the problem. ETA 2 hours.

[Resolved] Fedora Rawhide testing blocked on dnf5 breakage

Tue, 18 Jul 2023 08:00:00 +0000

Update

Rawhide is working again \o/.

Outage

The switch to dnf5 broke tmt. We are upgrading to tmt 1.25. We hope to resolve the problem today.

[Resolved] Fedora Rawhide testing blocked on dnf5

Thu, 22 Jun 2023 09:00:00 +0000

Resolved - All issues were addressed, Fedora-Rawhide testing works again \o/

Outage - The switch to dnf5 broke Testing Farm again. Some use cases might work though, no current ETA to resolve all the problems. We will be posting updates here.

[Resolved] Fedora Rawhide testing blocked on dnf5

Wed, 21 Jun 2023 12:30:00 +0000

Resolved - All issues were addressed, Fedora-Rawhide testing works again \o/

Outage - The switch to dnf5 broke Testing Farm. We are trying to hotpatch the problems to bring back testing against Rawhide.

[Resolved] Red Hat Ranch affected by RH infra outage

Thu, 01 Jun 2023 10:30:00 +0000

Resolved - Outage was resolved, all affected builds restarted.

Outage - Testing might fail due to internal RH infrastructure having a planned outage. ETA 1 day.

[Resolved] Expired Fedora-Rawhide images in Public Ranch

Mon, 29 May 2023 02:00:00 +0000

Resolved - Outage was resolved, failed jobs will require restart.

Outage - Images for Fedora Rawhide have expired. We are refreshing it. ETA 30 min.

[Resolved] Unexpected outage due to new release 2023-04.1

Fri, 05 May 2023 02:00:00 +0000

Outage 2023-05-05 12:00 UTC - Outage was resolved.

Outage 2023-05-05 02:00 UTC - We are deploying new Testing Farm, and it is not going super well. Expect delays and in case of unexpected errors, please restart your jobs.

[Resolved] Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Tue, 02 May 2023 08:00:00 +0000

Resolved 2023-05-02 12:00 UTC - Problem was resolved. Queue was resolved, requests should start resuming, no issues are expected.

Outage 2023-05-02 08:00 UTC - We stopped BaseOS CI and Testing Farm Red Hat ranch, because we see Artemis provisioning c5.24xlarge machines, we are investigation why. This is causing elevated costs.

[Resolved] Public Ranch - unexpected outage

Fri, 28 Apr 2023 04:00:00 +0000

Resolved 2023-04-28 10:00 UTC - Composes were updated. We are resuming the queue. If you encountered some errors, please retry your testing.

Outage 2023-04-28 08:00 UTC - There is an outage in public ranch, we need to update composes. Queue is paused. ETA 2 hours. Failed tests will need restart afterwards.

[Resolved] Public Ranch - unexpected outage

Tue, 18 Apr 2023 14:00:00 +0000

Resolved 2023-04-18 20:00 UTC - Outage was resolved, it can take some time to process all requests.

Ongoing 2023-04-18 19:00 UTC - Unfortunately, it is taken even longer as expected. ETA hopefully today.

Ongoing 2023-04-18 16:00 UTC - Unfortunately, it is taken longer as expected. ETA another 2 hours.

Outage 2023-04-18 14:00 UTC - There is an outage in public ranch, we need to redeploy. ETA 2 hours.

[Resolved] Public Ranch - expired images

Fri, 14 Apr 2023 03:00:00 +0000

Resolved 2023-04-14 09:00 UTC - Images were expired, problem should be resolved.

Outage 2023-04-14 03:00 UTC -There is an outage in public ranch, we are looking into it.

[Resolved] CentOS Stream 9 aarch64 broken in Public Ranch

Fri, 14 Apr 2023 00:00:00 +0000

Resolved 2023-04-17 20:00 UTC - Image was updated to a working one, issue is resolved \o/

Outage 2023-04-14 00:00 UTC - The CentOS Stream 9 aarch64 image is broken, all jobs will fail with GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial in pipeline.log. We are waiting for the image to refresh, until then all jobs will fail.

[Resolved] Red Hat Ranch - deploying 2023-03.1 release - large queue expected

Thu, 06 Apr 2023 10:00:00 +0000

Notice 2023-04-06 14:00 UTC - Testing Farm 2023-03.1 was deployed to Red Hat ranch \o/

Notice 2023-04-06 10:00 UTC - we deploying Testing Farm 2023-03.1 to Red Hat ranch, larger queue is expected.

[Resolved] Red Hat Ranch - provisioning outage

Mon, 03 Apr 2023 08:00:00 +0000

Outage 2023-04-03 12:00 UTC - we believe the issue was resolved, please restart your jobs if needed.

Outage 2023-04-03 08:00 UTC - we are experiencing outage of our provisioner. It is expected some requests will end in error. Also we have a larger queue, so longer test times expected.

[Resolved] Red Hat Ranch - larger queue

Wed, 29 Mar 2023 08:00:00 +0000

Resolved 2023-03-29 10:00 UTC - Queue is back to normal, resolving.

Outage 2023-03-29 08:00 UTC - There was a large queue in the morning, we scaled the deployments and it should be processed soon.

[Resolved] Public ranch - Artemis resource ended in 'error' state

Tue, 28 Mar 2023 20:00:00 +0000

Outage 2023-03-29 20:00 UTC - Issues were resolved.

Outage 2023-03-28 20:00 UTC - We are seeing some provisioning instability, we believe we addressed the problem, please restart your jobs if needed.

[Resolved] CentOS Stream 8 broken in Public Ranch

Thu, 16 Mar 2023 08:00:00 +0000

Resolved 2023-03-20 11:00 UTC - The problem was resolved. CentOS Stream 8 changed the naming in AWS, so we were using old image. Seems affected tests are now passing \o/.

Outage 2023-03-16 08:00 UTC - We are seeing various issues with the updated CentOS Stream 8 in CI. This is due to the update to RHEL-8.7.0 bits we believe. More information will land once we analyze the problems. The requests uusually error out unexpectedly.

Some example issues:

https://artifacts.dev.testing-farm.io/d5256819-3f06-4dc4-bf01-923c135de16a/

[Resolved] Fedora-Rawhide broken in Red Hat Ranch

Fri, 10 Mar 2023 08:00:00 +0000

Resolved 2023-03-16 12:00 UTC - We have found the issue, Rawhide is broken on Xen instances. Filled to Fedora infrastructure: https://pagure.io/fedora-infrastructure/issue/11185. We workarounded by using nitro instances for the testing.

Outage 2023-03-10 08:00 UTC - Fedora Rawhide is broken in Red Hat ranch and BaseOS CI. The instance does not boot. Requests will error out. Currently the root cause is unknown and not ETA for resolving.

[Resolved] Red Hat internal infrastructure outage

Fri, 10 Feb 2023 09:00:00 +0000

Outage 18:00 UTC 11th Feb - All processing resumed, long queue for next few hours.

Outage 16:00 UTC - Requests to Red Hat ranch might fail unexpectedly due to the internal outage. The RHEL images depend on some of the infrastructure being down - namely repositories to beakerlib and such. The outage should last until Fri, 10 Feb 2023 22:00:00 UTC.

Outage 09:00 UTC - We paused processing of all jobs via BaseOS CI, to mitigate outage effects.

[Resolved] BaseOS CI - large queue, expect delays.

Mon, 30 Jan 2023 16:00:00 +0000

📓 Notice 2023-02-13 18:00:00 UTC - We believe things are now a lot better, we are able to keep with the queue. Resolving.

📓 Notice 2023-01-30 16:00:00 UTC - We are experiencing a larger queue on BaseOS CI. This can affect baseos-ci.* and leapp.* tests. Delays are expected. We are having trouble scaling the deployment due to some bugs in the deployment :(

[Resolved] Public Ranch - large queue, expect delays.

Wed, 25 Jan 2023 10:00:00 +0000

Resolved 2023-01-25 17:00:00 UTC - Whole queue was processed. No more delays are expected.

Notice 2023-01-25 15:00:00 UTC - We are about 60% done, workers scaled, we expect 1-2 hours to process all the queues.

Notice 2023-01-25 10:00:00 UTC - We are rerunning 1.2k jobs due to the outage. It might take some time to process.

[Resolved] Public Ranch outage - requests stuck in queued

Tue, 24 Jan 2023 15:00:00 +0000

Resolved 2023-01-25 10:00:00 UTC - Issue was resolved. All affected users should retry testing against Public ranch.

Outage 2023-01-24 15:00:00 UTC - Due to a deployment change all requests stuck in queued state.

[Resolved] BaseOS CI Outage

Tue, 06 Dec 2022 08:00:00 +0000

Resolved 17:00 UTC - The issue was resolved. We are restarting all affected jobs. It might take some time for all jobs to process.

Outage 15:00 UTC - We still do not have an ETA here, if the issue would persist, we will workaround it. Expecting to resume jobs this night.

Outage 12:00 UTC - We paused processing of all jobs, to mitigate outage effects and discussing how to mitigate this problem in the future.

Outage 08:00 UTC - BaseOS CI is currently failing some of it’s test jobs due to an internal outage. More info in TFT-1770.

[Resolved] Red Hat Ranch outage - tests failing [Errno -2] Name or service not known

Wed, 30 Nov 2022 09:00:00 +0000

Resolved 16:40 UTC - Outage was resolved. There were multiple issues here - incorrect DNS settings, code sensitive on DNS resolution, podman not using host networking.

Update 14:40 UTC - We are testing a woraround to be able to enable back the service. Fingers crossed.

Update 13:40 UTC - Seems we found a setup which makes the DNS resolving more reliable (removing aadvark-dns) and a bug from the latest release, making this error not being retried. As for why it worked before ….

Reopened 11:45 UTC - The IT DNS was a false lead. Seems we are hitting some podman DNS bug. Investigation continues.

Resolved 11:45 UTC - As it turned out, the VPC we used for years was using a DNS setup that can be prone to problems. After migrating to another VPC the problem is gone.

Update 10:30 UTC - The problems have been identified, currently it seems IT DNS problems causing random resolution errors. We are testing a workaround and working with IT to resolve the problem.

Outage 8:00 UTC - We are investigating some DNS problems on the workers. All tests might fail with Failed to establish a new connection: [Errno -2] Name or service not known'.

[Resolved] Red Hat Ranch outage - artifacts storage empty

Tue, 29 Nov 2022 08:00:00 +0000

Resolved 10:30 UTC - The issue was addressed via https://gitlab.com/testing-farm/infrastructure/-/merge_requests/102. Note that all affected jobs wil need a restart.

Update 9:00 UTC - We have identified the problem and are working on a fix.

Outage 8:00 UTC - The artifacts.osci.redhat.com is empty after run finished executing. This is due to a bad change in the artifacts syncing code. WE are working on a fix.

[Resolved] Public Ranch Outage - artifacts storage out of inodes

Tue, 22 Nov 2022 14:00:00 +0000

Follow-up - It might be needed to restart your jobs, they might have failed without logs, as the artifacts were not correctly archived.

Resolved 20:00 UTC - The issue was identified and workarounded. We got out of inodes on the 10TB storage. Follow up in TFT-1735.

Outage 14:00 UTC - The artifacts.dev.testing-farm.io is out of inodes, we are looking into cleaning it up and understanding why it happened.

[Resolved] BaseOS CI Outage - UMB message sending

Mon, 31 Oct 2022 08:00:00 +0000

Resolved 12:00 UTC - The issue was resolved, the UMB cert was rotated. We are restarting all affected jobs. Processing the queue can take some time. Contact us in case of trouble.

In Progress 10:30 UTC - The issue was identified - an expired UMB certificate. We are working with IT to be able to rotate it. Current ETA is at least 2 more hours.

Outage 8:00 UTC - BaseOS CI is not processing any jobs currently, it is failing to send out any UMB message. We are investigating the problem.

[Resolved] Red Hat Ranch is disabled due to a wide Red Hat infrastructure outage

Mon, 10 Oct 2022 08:00:00 +0000

Outage - Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch is disabled. Currently there is no ETA for resolving the problem.

[Resolved] Public Ranch outage - Copr API v2 is deprecated

Thu, 22 Sep 2022 06:00:00 +0000

Resolved - Issue was hotfixed on all ranches via https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/223.

Update 13:00 - To mitigate consequences of the outage, whole Public Ranch was suspended.

Outage 08:00 - Copr deprecated API v2 today. We are working on migration to API v3. This affects all requests which test copr builds. All requests from Packit are affected.

[Resolved] BaseOS CI Outage

Fri, 12 Aug 2022 18:00:00 +0000

Resolved - The issue was resolved by dropping the requirement for the bits stored in BRQ.

Outage - BaseOS CI is not processing any jobs currently, as it is broken due to a dependency from BRQ, where the LAB is being moved. We are working hard to get it back to processing jobs.

[Resolved] CentOS Stream Koji Outage

Thu, 11 Aug 2022 08:00:00 +0000

Resolved The issue was identified and resolved by the CentOS Stream infrastructure team.

Outage 2022-08-11 08:00 UTC - The CentOS Stream Koji is down https://lists.centos.org/pipermail/centos-devel/2022-August/120524.html. No ETA available currently.

[Resolved] Cyborg Development Cluster outage

Mon, 01 Aug 2022 11:00:00 +0000

Update - Cluster was reinstalled.

Outage - The Cyborg Development Cluster is currently down, seems due to PSI outage during weekend, we are reinstalling it.

[Resolved] CentOS Stream 9 repositories broken in CI

Thu, 16 Jun 2022 09:00:00 +0000

Updated 2022-06-16 15:00:00 UTC - Things look back to normal. Resolving the outage.

Updated 2022-06-16 10:00:00 UTC - The underlying infra problem had been fixed alreday by CentOS infra team, but it might take some time for mirrors to pick up the fixes. More info in https://pagure.io/centos-infra/issue/812. Seems the issues is affecting only Cs9, other supported composes should not be affected. We will be monitoring the problem and update the status once things settle.

Outage 2022-06-16 9:00:00 UTC - We are seeing a lot of errors from CentOS Stream 9 related to broken hashes in repositories. The root cause is currently unknown, reported to centos-infra in https://pagure.io/centos-infra/issue/812

[Resolved] Cyborg Development Cluster credentials outage

Thu, 02 Jun 2022 08:00:00 +0000

Resolved - Problem was resolved by reinstalling the cluster.

Outage - The Cyborg Development Cluster credentials are invalid. We are looking into the problem. As a workaround follow https://issues.redhat.com/browse/TFT-1338.

[Resolved] Covscan testing blocked on BaseOS CI

Fri, 25 Mar 2022 12:00:00 +0000

Resolved - covscan testing on BaseOS CI was resumed. Outage is over.

Outage - covscan is boggled with too many scans from Errata due to a known issue. We will be blocking testing until covscan is freed up and can peform CI testing. Currently there is no ETA.

[Resolved] rpminspect fails with "internal libpod error"

Tue, 08 Feb 2022 12:00:00 +0000

Resolved 2022-04-09 - Issue was resolved by updating to a podman pre-release which is fixing the bug.

Update 2022-03-25 - Upstream has a fix for the problem. We are waiting for a custom module build with the fix, and we will update our workers with it.

Update 2022-02-25 - We have found a minimal reproducer - https://github.com/containers/podman/issues/13227#issuecomment-1050997458

Update 2022-02-25 - Investigation is still ongoing, we are not able to reproduce the problem manually. We will try today to update to podman 4.0.1 to see if it helps with the issue.

Update 2022-02-14 - Issue was filled to podman - https://github.com/containers/podman/issues/13227

Investigating - We’re aware of an issue affecting rpminspect and causing some of the tests blow up on errors like “Error: timed out waiting for file /var/lib/containers/storage/overlay-containers/4dbddac31d1a5529d055922e3fca8d200585b5ca562e7eb9976f32d7dd96b8f2/userdata/1e05be2210ea6433b84bba70c720b55586d4987fde7005428492e583d8ebc3a7/exit/4dbddac31d1a5529d055922e3fca8d200585b5ca562e7eb9976f32d7dd96b8f2: internal libpod error”

[Resolved] Fedora CI outage - AWS images are gone

Sat, 05 Feb 2022 20:00:00 +0000

Update - We uploaded our own images to overcome the problems with Fedora Infrastructure. The outage is resolved. Please restart jour tests jobs if you need them.

Update - Because still no ETA on fixing Fedora CI provided AWS images, we are trying to upload them to AWS ourselves.

Update - We found out all images are gone after the weekend from Fedora AWS - https://pagure.io/fedora-infrastructure/issue/10532. We are working with Fedora admins to get things in good shape, but there is currently no ETA as we might need to wait for US folks to come to work.

[Resolved] Broken rpmdeplint and rpminspect in Fedora CI

Mon, 31 Jan 2022 22:00:00 +0000

Update - The underlying problem was fixed. Rpminspect and RPMdeplint works again.

Update - We are working on fixing the problem, it was broken with our migration from CentOS-8 to CentOS-Stream-8.

Investigating - We’re aware of an issue affecting rpmdeplint and rpminspect on Public Ranch.

[Resolved] Large queue times on Red Hat Ranch

Thu, 27 Jan 2022 10:30:00 +0000

Update 2022-02-25 - Testing Farm was scaled and should have enough capacity for the current workload.

Update 2022-02-01 - Currently the load is ok, but we are still working on a more permanent solution

Update - We are working on a fix to scale the ranch to handle the load.

Investigating - We’re aware of an issue affecting queue times on Red Hat Ranch of Testing Farm. Testing can be delayed several hours. It is mostly affecting rpmdeplint and rpminspect.

[Resolved] Large queue times on Public Ranch

Mon, 03 Jan 2022 17:00:00 +0000

Update - We now have better scallable workers on Public Ranch and they are dynamic and detected from the deployed infrastructure provider \o/

Update - We finally fixed all the issues with the new deployment and are scaling to cope up with the 1.6k+ jobs queued up. Things should be better in few hours!

Investigating - We’re aware of an issue affecting queue times on Public Ranch of Testing Farm. Testing can be delayed several hours. Some old jobs might have failed. We are scaling the infrastructure to better cope with the high load.

Known issues

Mon, 03 Jan 2022 00:17:00 +0000

There are currently no known issues, if you have troubles, contact us :)

Recently resolved

Rawhide image update is blocked on BZ#2034336 - resolved on 2021-01-03 00:12:00 - seems underlying problems have been resolved finally

[Resolved] Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Wed, 29 Dec 2021 11:00:00 +0000

Update - Images were updated, failed tests restarted. In case you encounter issues, please restart your failed jobs.

Update - Red Hat tracking issue

Investigating - We’re aware of expired images for RHEL-8.6.0 and RHEL-9.0.0. They are causing failures for installability test and all functional tests. We are working to resolve the problem ASAP and will try to restart all affected jobs once the issue is resolved.

[Resolved] Issues with provisioning in Red Hat Ranch

Tue, 21 Dec 2021 16:00:00 +0000

Update - Issue was reported to Artemis, we will continue investigation there.

Update - We resolved the problem by restarting the worker. We are preparing an issue to Artemis to follow up. Looks like in this case the worker was not able to reconnect to the rabbitmq for whatever reason. The outage is cleared, one more update to the issue will follow once we report the problem.

Investigating - We’re aware of an issue affecting provisioning of machines on Red Hat Ranch of Testing Farm. Seems the problem is with our Artemis provisioning. We will be updating this issue once we know more. All queues are paused, no requests should be lost for now.

[Resolved] Issues Testing Fedora Rawhide

Sun, 19 Dec 2021 20:00:00 +0000

Update - The incident has been resolved. Root cause of the problem is Bug 2034336. We rolled back to Fedora Rawhide image from 2021-12-12 which does not contain the issue.

Investigating - We’re aware of an issue affecting all testing on Fedora Rawhide. It appeared as we upgraded to the latest image 2021-12-17.