Fedora CI on Testing Farm Status

Rpminpsect on Fedora-CI PRs are failing when trying to download a scratch-build

Fri, 15 May 2026 08:39:00 +0000

Outage | 2026-05-15 08:39:00 UTC

https://forge.fedoraproject.org/ci/tickets/issues/560

[Resolved] Large queue on public ranch

Wed, 22 Apr 2026 10:17:00 +0000

Outage | 2026-04-22 10:17:00 UTC

The queue on public ranch has exploded, we are trying to scale up.

Update | 2026-04-22 23:44:00 UTC

Queues are back to normal. We were able to scale up, so we should have a bit more firepower here.

Resolved | 2026-04-22 23:45:00 UTC

Resolving

[Resolved] Elevated API errors

Mon, 20 Apr 2026 15:19:00 +0000

Outage | 2026-04-20 15:19:00 UTC

We are seeing elevated API errors, we are investigating the issue.

Update | 2026-04-20 16:14:00 UTC

This should be better now.

Resolved | 2026-04-20 17:52:00 UTC

Resolving, the issue is gone

[Resolved] Fedora Rawhide and ELN composes provisioning errors out

Wed, 08 Apr 2026 14:53:00 +0000

Outage | 2026-04-08 14:53:00 UTC

Testing-farm jobs using the Fedora-ELN and Rawhide composes fail to provision or fail during the excution of the pre-artifact-installation playbook.

As far as we can tell it goes through something like: image booted, testing farm connected to it, started running playbooks, got to the tasks which upgrades all the rpms and reboots, could not connect back due to emergency mode.

Update | 2026-04-17 01:45:00 UTC

Also hitting Fedora-Rawhide now. Seems like some bug from Rawhide, investigation is ongoing.

Resolved | 2026-05-08 02:54:00 UTC

This is resolved via workarounds. Fedora-eln is now not updated to latest bits.

[Resolved] Testing Farm container based testing is broken

Wed, 25 Mar 2026 15:00:00 +0000

Outage | 2026-03-25 15:00:00 UTC

We are sorry, the rotation of secrets broken our quay.io creds.

Resolved | 2026-03-26 08:57:00 UTC

Resolved. Testing Farm container testing is back.

[Resolved] Testing Farm 2026-03.2 being deployed

Thu, 19 Mar 2026 19:18:00 +0000

Notice | 2026-03-19 19:18:00 UTC

We are deploying new Testing Farm version with tmt-1.69, stay tuned. Small downtime of API is expected, otherwise no other issues should arise. We will update this status entry if needed. ETA 4 hours.

Update | 2026-03-20 16:02:00 UTC

There was a small bug failing ~20 jobs in RH ranch causing Ansible to fail with: The conditional check ‘ansible_pkg_mgr in [‘unknown’, ‘dnf’]’ failed

Update | 2026-03-21 22:40:00 UTC

Testing Farm is now deployed, we will send out the release notes on Monday.

Update | 2026-03-21 22:42:00 UTC

There was an issue in Ansible setup for Fedora in Red Hat ranch, some of the jobs have failed, please restart them if you encountered Could not import the libdnf5 python module using /var/opt/standalone-python/bin/python3.

Resolved | 2026-03-23 19:02:00 UTC

Testing Farm was deployed! Enjoy!

[Resolved] Testing Farm outage - Cannot execute the rule

Thu, 05 Mar 2026 21:00:00 +0000

Outage | 2026-03-05 21:00:00 UTC

Both ranches are suffering outage after the most recent release deployment. A mistake in configuration is causing pipelines to fail. We are currently investigating why the corresponding fix have not been deployed properly.

Update | 2026-03-06 10:33:00 UTC

It was affecting Public Ranch too. Please restart your failed requests. The issue was fixed by reverting change. The issue was not affecting requests testing against containers.

Resolved | 2026-03-06 10:00:00 UTC

Hotfix deployed and new request are working.

[Resolved] Testing Farm 2026-02.2 was deployed on 2026-02-19

Thu, 19 Feb 2026 18:58:00 +0000

Outage | 2026-02-19 18:58:00 UTC

New Testing Farm version was deployed, but we will send out release notes on Monday, as we did not make it. For a previous of release notes see: https://testing-farm.gitlab.io/-/docs/root/-/jobs/13185451238/artifacts/preview/Testing%20Farm/0.1/releases/2026-02.2.html

Resolved | 2026-02-23 10:21:00 UTC

Resolving, new release is deployed fully.

[Resolved] Fedora CI jobs on bodhi updates are not triggered

Tue, 17 Feb 2026 13:10:00 +0000

Outage | 2026-02-17 13:10:00 UTC

Something is wrong on our side and the messaging bus. We are trying to get this resolved.

Resolved | 2026-02-18 14:25:00 UTC

Issue resolved, there will be a big job queue while it catches up.

[Resolved] Testing-farm jobs on rawhide fail signature check

Tue, 17 Feb 2026 12:14:00 +0000

Outage | 2026-02-17 12:14:00 UTC

Testing-farm jobs that require artifacts (for example scratch builds) are now failing on rawhide due to the signature enforcement proposal. We are investigating if this is an issue on our side or the change proposal.

For more info, follow https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BYG3JSPRB3R2YQWP6O3ZD2GMNFRVM7JS/#R2422WIXVQYIBCAXHHTWG7ITI2P6PERA

[Resolved] Large queue on public ranch due to provisioning outage

Mon, 16 Feb 2026 09:41:00 +0000

Outage | 2026-02-16 09:41:00 UTC

We just recovered from a provisioning outage on public ranch. Things should start resuming, but because if limited scalability it might take a while (1-2 hours)

Resolved | 2026-02-16 11:11:00 UTC

The queue is gone, outage resolved.

[Resolved] Fedora 44 not available in Testing Farm

Mon, 09 Feb 2026 11:46:00 +0000

Outage | 2026-02-09 11:46:00 UTC

The import is in progress after branching. We are working it out. ETA later today.

Resolved | 2026-02-10 02:45:00 UTC

Fedora 44 is now available.

[Resolved] High queue on public ranch, expect delays

Thu, 05 Feb 2026 11:00:00 +0000

Outage | 2026-02-05 11:00:00 UTC

We are experiencing high load on Testing Farm. Investigation in place. Expect delays.

Update | 2026-02-05 15:00:00 UTC

The situation is ongoing, current average queue time is 3.5 hours

Update | 2026-02-06 01:40:00 UTC

Things are improving, we had to kill testing from scap via atex. Seems like some new integration using reservations is not cleaning up resources, causing all the requests to stay up long and take all resources, blocking other users.

Update | 2026-02-06 02:00:00 UTC

queue is almost gone, we are monitoring the situation

Resolved | 2026-02-09 10:26:00 UTC

Problems should be gone as of last week. The misusage was addressed on the user side.

[Resolved] Testing Farm 2026-02.1 deployment in progress

Mon, 02 Feb 2026 10:46:00 +0000

Outage | 2026-02-02 10:46:00 UTC

We are deploying a new TF release, brief outages of API might happen during this time.

Update | 2026-02-02 18:51:00 UTC

Deployment is half done, and ongoing to the next day

Resolved | 2026-02-04 15:59:00 UTC

Testing Farm 2026-02.1 is fully deployed. Resolving.

[Resolved] Elevated API errors in Testing Farm

Mon, 19 Jan 2026 09:00:00 +0000

Outage | 2026-01-19 09:00:00 UTC

We are experiecing read timeouts on our API. Issue is under investigation.

Update | 2026-01-19 20:00:00 UTC

We are patching API to better recycle stale connections

Update | 2026-01-19 20:21:00 UTC

Issue should be addressed, we are monitoring the situation. https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 20:34:00 UTC

Things are back to stable, we hotpatched production with: https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 21:11:00 UTC

The issues is back :( we are looking further

Resolved | 2026-01-19 23:33:00 UTC

Resolving, we did nto see this anymore after applying the fixes.

[Resolved] High koji build queue for ci channel

Fri, 16 Jan 2026 13:25:00 +0000

Outage | 2026-01-16 13:25:00 UTC

Fedora mass rebuild is ongoing and it has a higher priority than the ci channel for Fedora CI scratch builds. Expect a long delay to both status updates on PRs as well as scratch builds not starting.

Resolved | 2026-01-19 09:56:00 UTC

Mass rebuild build queue passed. Scratch builds queue has returned to normal

[Resolved] Public Ranch provisioning unstable, high queue

Wed, 12 Nov 2025 13:00:00 +0000

Outage | 2025-11-12 08:00:00 UTC

We are seeing issues with provisioning in Public ranch. Investigation is in progress.

Resolved | 2025-11-16 20:47:00 UTC

Resolving, after updating to Testing Farm v0.0.82 with a fix seems provisioning is stable.

[Resolved] quay.io is down

Mon, 20 Oct 2025 07:51:00 +0000

Outage | 2025-10-20 07:51:00 UTC

We are aware of quay.io outage, we are gatheting more information. This will affect our services, but the real extent is not yet known.

Update | 2025-10-20 07:58:00 UTC

Seems connected to the AWS ouatage in us-east-1: https://health.aws.amazon.com/health/status

Update | 2025-10-20 09:10:00 UTC

Outage was resolved by routing traffic to us-east-2.

Resolved | 2025-10-20 09:10:00 UTC

Resolving

[Resolved] Fedora CI not triggering except bodhi tests

Tue, 07 Oct 2025 22:00:00 +0000

Outage | 2025-10-07 22:00:00 UTC

Fedora CI triggers are broken, everything excpect bodhi functional tests won’t be run. We are trying to resolve, but we need OSCI to help. This includes rpminspect, rpmdeplint, installability and dist-git PR testing.

Update | 2025-10-08 23:25:00 UTC

Tracked internally https://redhat-internal.slack.com/archives/C04L304KVH9/p1759965536710569

Resolved | 2025-10-09 08:40:00 UTC

Issue was resolved by OSCI team.

[Resolved] Provisioning errors in Public Ranch due to an outage

Mon, 01 Sep 2025 16:00:00 +0000

Outage | 2025-09-01 16:00:00 UTC

The newly created public workers did not have access to Artemis causing provisioning errors during 18:00 UTC to ~ midnight.

Update | 2025-09-02 00:53:00 UTC

The problem was addressed and things should be getting back to normal. Please restart your failed tests.

Resolved | 2025-09-02 17:27:00 UTC

Resolved

[Resolved] Fedora provisioning errors

Tue, 19 Aug 2025 17:23:00 +0000

Outage | 2025-08-19 17:23:00 UTC

We are seeing elevated provisioning issues on Fedora Rawhide mostly. The issues is under investigation.

Update | 2025-08-26 17:53:00 UTC

The issue was found, it is caused by a bug in cloud-init causing wrong ordering of services at startup, breaking in some ocassions sshd. See https://bugzilla.redhat.com/show_bug.cgi?id=2390898 for details. We are waiting for Fedora-Rawhide and Fedora-43 fixed builds to land. Hopefully that will stabilize the testing.

Resolved | 2025-09-01 11:49:00 UTC

This is finally resolved. Both Rawhide and F43 should be stable now.

[Resolved] Fedora CI Jenkins is down

Mon, 18 Aug 2025 08:39:00 +0000

Outage | 2025-08-18 08:39:00 UTC

We are investigating the root cause. No ETA yet.

Update | 2025-08-18 09:10:00 UTC

The Jenkins is back, but it is processing all the requests, can take some time.

Update | 2025-08-18 09:56:00 UTC

We believe Jenkins is processing stuff, but it is overloaded, so expect delays. We will be monitoring the situation and report back here.

Update | 2025-08-18 14:17:00 UTC

Unfortunately the web interface takes too long to load, but the Jenkins is alive. We are trying to scale up the deployment to be able to deal with this load. ETA 1h.

Update | 2025-08-18 17:45:00 UTC

We bumped the memory (8 -> 24Gi) and CPU (3 -> 6) for the instance.

Update | 2025-08-18 18:18:00 UTC

Jenkins is back, UI is responsive, queue is large but going down. We will be monitoring the situation for the next hours. Please be patient. We know that there will be some failed jobs resulting as missing tests in Bodhi and Pagure. Feel free to restart all jobs if you are in hurry.

Resolved | 2025-08-18 20:13:00 UTC

Job queue has finished processing. No obvious issues (apart from the existing ones) were found. If you find anything amiss let us know on the #fedora-ci channel.

[Resolved] Fedora Infra outage affecting koji and dist-git

Mon, 04 Aug 2025 06:00:00 +0000

Outage | 2025-08-04 06:00:00 UTC

We are aware of outage of Fedora Infrastructure. The testing will be blocked or delayed. No ETA yet, we are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Resolved | 2025-08-05 08:29:00 UTC

The outage is resolved, Fedora infra should be back.

[Resolved] src.fedoraproject.org is down

Thu, 26 Jun 2025 10:00:00 +0000

Outage | 2025-06-26 10:00:00 UTC

DC reloctation outage expected. Pleases see status here https://status.fedoraproject.org/ CentOS pkg repos are also affected.

Resolved | 2025-07-14 07:53:00 UTC

The outage is resolved. All Fedora systems are back to normal. Resolving outage.

[Resolved] src.fedoraproject.org down

Tue, 17 Jun 2025 11:11:00 +0000

Outage | 2025-06-17 10:00:00 UTC

src.fedoraproject.org is down and any jobs relying on this resource will fail. Please contact the fedora team about this.

Resolved | 2025-06-17 11:23:00 UTC

Site back online.

[Resolved] IT resilience testing multiple SP services down

Fri, 13 Jun 2025 13:00:00 +0000

Outage | 2025-06-13 13:00:00 UTC

tps-server

Resolved | 2025-06-13 21:00:00 UTC

Testing complete

[Resolved] ansible-core breaks ansible-lint

Tue, 20 May 2025 07:42:00 +0000

Outage | 2025-05-20 07:42:00 UTC

Tooling affected by https://github.com/ansible/ansible-lint/issues/4611

Update | 2025-08-07 12:00:00 UTC

This has been mostly mitigated by forcing the ansible version to something before 2.19.

Update | 2025-08-18 09:12:00 UTC

Seems resolved for some time now.

Resolved | 2025-08-19 17:20:00 UTC

Resolving

[Resolved] Fedora CI is not triggering on events due to expired Fedora message bus

Wed, 18 Dec 2024 09:00:00 +0000

Resolved 2024-12-18 15:00:00 UTC - The certificate was rotated.

Outage 2024-12-18 09:00:00 UTC - Fedora CI is not triggering on events, we are trying to resolve the outage. The certificate for message bus has expired today.