Fedora CI / installability on Testing Farm Status

[Resolved] Large queue on public ranch

Wed, 22 Apr 2026 10:17:00 +0000

Outage | 2026-04-22 10:17:00 UTC

The queue on public ranch has exploded, we are trying to scale up.

Update | 2026-04-22 23:44:00 UTC

Queues are back to normal. We were able to scale up, so we should have a bit more firepower here.

Resolved | 2026-04-22 23:45:00 UTC

Resolving

[Resolved] Elevated API errors

Mon, 20 Apr 2026 15:19:00 +0000

Outage | 2026-04-20 15:19:00 UTC

We are seeing elevated API errors, we are investigating the issue.

Update | 2026-04-20 16:14:00 UTC

This should be better now.

Resolved | 2026-04-20 17:52:00 UTC

Resolving, the issue is gone

[Resolved] Fedora Rawhide and ELN composes provisioning errors out

Wed, 08 Apr 2026 14:53:00 +0000

Outage | 2026-04-08 14:53:00 UTC

Testing-farm jobs using the Fedora-ELN and Rawhide composes fail to provision or fail during the excution of the pre-artifact-installation playbook.

As far as we can tell it goes through something like: image booted, testing farm connected to it, started running playbooks, got to the tasks which upgrades all the rpms and reboots, could not connect back due to emergency mode.

Update | 2026-04-17 01:45:00 UTC

Also hitting Fedora-Rawhide now. Seems like some bug from Rawhide, investigation is ongoing.

Resolved | 2026-05-08 02:54:00 UTC

This is resolved via workarounds. Fedora-eln is now not updated to latest bits.

[Resolved] Testing Farm 2026-03.2 being deployed

Thu, 19 Mar 2026 19:18:00 +0000

Notice | 2026-03-19 19:18:00 UTC

We are deploying new Testing Farm version with tmt-1.69, stay tuned. Small downtime of API is expected, otherwise no other issues should arise. We will update this status entry if needed. ETA 4 hours.

Update | 2026-03-20 16:02:00 UTC

There was a small bug failing ~20 jobs in RH ranch causing Ansible to fail with: The conditional check ‘ansible_pkg_mgr in [‘unknown’, ‘dnf’]’ failed

Update | 2026-03-21 22:40:00 UTC

Testing Farm is now deployed, we will send out the release notes on Monday.

Update | 2026-03-21 22:42:00 UTC

There was an issue in Ansible setup for Fedora in Red Hat ranch, some of the jobs have failed, please restart them if you encountered Could not import the libdnf5 python module using /var/opt/standalone-python/bin/python3.

Resolved | 2026-03-23 19:02:00 UTC

Testing Farm was deployed! Enjoy!

[Resolved] Testing Farm outage - Cannot execute the rule

Thu, 05 Mar 2026 21:00:00 +0000

Outage | 2026-03-05 21:00:00 UTC

Both ranches are suffering outage after the most recent release deployment. A mistake in configuration is causing pipelines to fail. We are currently investigating why the corresponding fix have not been deployed properly.

Update | 2026-03-06 10:33:00 UTC

It was affecting Public Ranch too. Please restart your failed requests. The issue was fixed by reverting change. The issue was not affecting requests testing against containers.

Resolved | 2026-03-06 10:00:00 UTC

Hotfix deployed and new request are working.

[Resolved] Testing Farm 2026-02.2 was deployed on 2026-02-19

Thu, 19 Feb 2026 18:58:00 +0000

Outage | 2026-02-19 18:58:00 UTC

New Testing Farm version was deployed, but we will send out release notes on Monday, as we did not make it. For a previous of release notes see: https://testing-farm.gitlab.io/-/docs/root/-/jobs/13185451238/artifacts/preview/Testing%20Farm/0.1/releases/2026-02.2.html

Resolved | 2026-02-23 10:21:00 UTC

Resolving, new release is deployed fully.

[Resolved] Large queue on public ranch due to provisioning outage

Mon, 16 Feb 2026 09:41:00 +0000

Outage | 2026-02-16 09:41:00 UTC

We just recovered from a provisioning outage on public ranch. Things should start resuming, but because if limited scalability it might take a while (1-2 hours)

Resolved | 2026-02-16 11:11:00 UTC

The queue is gone, outage resolved.

[Resolved] Fedora 44 not available in Testing Farm

Mon, 09 Feb 2026 11:46:00 +0000

Outage | 2026-02-09 11:46:00 UTC

The import is in progress after branching. We are working it out. ETA later today.

Resolved | 2026-02-10 02:45:00 UTC

Fedora 44 is now available.

[Resolved] High queue on public ranch, expect delays

Thu, 05 Feb 2026 11:00:00 +0000

Outage | 2026-02-05 11:00:00 UTC

We are experiencing high load on Testing Farm. Investigation in place. Expect delays.

Update | 2026-02-05 15:00:00 UTC

The situation is ongoing, current average queue time is 3.5 hours

Update | 2026-02-06 01:40:00 UTC

Things are improving, we had to kill testing from scap via atex. Seems like some new integration using reservations is not cleaning up resources, causing all the requests to stay up long and take all resources, blocking other users.

Update | 2026-02-06 02:00:00 UTC

queue is almost gone, we are monitoring the situation

Resolved | 2026-02-09 10:26:00 UTC

Problems should be gone as of last week. The misusage was addressed on the user side.

[Resolved] Testing Farm 2026-02.1 deployment in progress

Mon, 02 Feb 2026 10:46:00 +0000

Outage | 2026-02-02 10:46:00 UTC

We are deploying a new TF release, brief outages of API might happen during this time.

Update | 2026-02-02 18:51:00 UTC

Deployment is half done, and ongoing to the next day

Resolved | 2026-02-04 15:59:00 UTC

Testing Farm 2026-02.1 is fully deployed. Resolving.

[Resolved] Elevated API errors in Testing Farm

Mon, 19 Jan 2026 09:00:00 +0000

Outage | 2026-01-19 09:00:00 UTC

We are experiecing read timeouts on our API. Issue is under investigation.

Update | 2026-01-19 20:00:00 UTC

We are patching API to better recycle stale connections

Update | 2026-01-19 20:21:00 UTC

Issue should be addressed, we are monitoring the situation. https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 20:34:00 UTC

Things are back to stable, we hotpatched production with: https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 21:11:00 UTC

The issues is back :( we are looking further

Resolved | 2026-01-19 23:33:00 UTC

Resolving, we did nto see this anymore after applying the fixes.

[Resolved] Public Ranch provisioning unstable, high queue

Wed, 12 Nov 2025 13:00:00 +0000

Outage | 2025-11-12 08:00:00 UTC

We are seeing issues with provisioning in Public ranch. Investigation is in progress.

Resolved | 2025-11-16 20:47:00 UTC

Resolving, after updating to Testing Farm v0.0.82 with a fix seems provisioning is stable.

[Resolved] quay.io is down

Mon, 20 Oct 2025 07:51:00 +0000

Outage | 2025-10-20 07:51:00 UTC

We are aware of quay.io outage, we are gatheting more information. This will affect our services, but the real extent is not yet known.

Update | 2025-10-20 07:58:00 UTC

Seems connected to the AWS ouatage in us-east-1: https://health.aws.amazon.com/health/status

Update | 2025-10-20 09:10:00 UTC

Outage was resolved by routing traffic to us-east-2.

Resolved | 2025-10-20 09:10:00 UTC

Resolving

[Resolved] Fedora CI not triggering except bodhi tests

Tue, 07 Oct 2025 22:00:00 +0000

Outage | 2025-10-07 22:00:00 UTC

Fedora CI triggers are broken, everything excpect bodhi functional tests won’t be run. We are trying to resolve, but we need OSCI to help. This includes rpminspect, rpmdeplint, installability and dist-git PR testing.

Update | 2025-10-08 23:25:00 UTC

Tracked internally https://redhat-internal.slack.com/archives/C04L304KVH9/p1759965536710569

Resolved | 2025-10-09 08:40:00 UTC

Issue was resolved by OSCI team.

[Resolved] Fedora provisioning errors

Tue, 19 Aug 2025 17:23:00 +0000

Outage | 2025-08-19 17:23:00 UTC

We are seeing elevated provisioning issues on Fedora Rawhide mostly. The issues is under investigation.

Update | 2025-08-26 17:53:00 UTC

The issue was found, it is caused by a bug in cloud-init causing wrong ordering of services at startup, breaking in some ocassions sshd. See https://bugzilla.redhat.com/show_bug.cgi?id=2390898 for details. We are waiting for Fedora-Rawhide and Fedora-43 fixed builds to land. Hopefully that will stabilize the testing.

Resolved | 2025-09-01 11:49:00 UTC

This is finally resolved. Both Rawhide and F43 should be stable now.

[Resolved] Fedora CI Jenkins is down

Mon, 18 Aug 2025 08:39:00 +0000

Outage | 2025-08-18 08:39:00 UTC

We are investigating the root cause. No ETA yet.

Update | 2025-08-18 09:10:00 UTC

The Jenkins is back, but it is processing all the requests, can take some time.

Update | 2025-08-18 09:56:00 UTC

We believe Jenkins is processing stuff, but it is overloaded, so expect delays. We will be monitoring the situation and report back here.

Update | 2025-08-18 14:17:00 UTC

Unfortunately the web interface takes too long to load, but the Jenkins is alive. We are trying to scale up the deployment to be able to deal with this load. ETA 1h.

Update | 2025-08-18 17:45:00 UTC

We bumped the memory (8 -> 24Gi) and CPU (3 -> 6) for the instance.

Update | 2025-08-18 18:18:00 UTC

Jenkins is back, UI is responsive, queue is large but going down. We will be monitoring the situation for the next hours. Please be patient. We know that there will be some failed jobs resulting as missing tests in Bodhi and Pagure. Feel free to restart all jobs if you are in hurry.

Resolved | 2025-08-18 20:13:00 UTC

Job queue has finished processing. No obvious issues (apart from the existing ones) were found. If you find anything amiss let us know on the #fedora-ci channel.

[Resolved] Fedora Infra outage affecting koji and dist-git

Mon, 04 Aug 2025 06:00:00 +0000

Outage | 2025-08-04 06:00:00 UTC

We are aware of outage of Fedora Infrastructure. The testing will be blocked or delayed. No ETA yet, we are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Resolved | 2025-08-05 08:29:00 UTC

The outage is resolved, Fedora infra should be back.

[Resolved] src.fedoraproject.org is down

Thu, 26 Jun 2025 10:00:00 +0000

Outage | 2025-06-26 10:00:00 UTC

DC reloctation outage expected. Pleases see status here https://status.fedoraproject.org/ CentOS pkg repos are also affected.

Resolved | 2025-07-14 07:53:00 UTC

The outage is resolved. All Fedora systems are back to normal. Resolving outage.

[Resolved] IT resilience testing multiple SP services down

Fri, 13 Jun 2025 13:00:00 +0000

Outage | 2025-06-13 13:00:00 UTC

tps-server

Resolved | 2025-06-13 21:00:00 UTC

Testing complete

[Resolved] ansible-core breaks ansible-lint

Tue, 20 May 2025 07:42:00 +0000

Outage | 2025-05-20 07:42:00 UTC

Tooling affected by https://github.com/ansible/ansible-lint/issues/4611

Update | 2025-08-07 12:00:00 UTC

This has been mostly mitigated by forcing the ansible version to something before 2.19.

Update | 2025-08-18 09:12:00 UTC

Seems resolved for some time now.

Resolved | 2025-08-19 17:20:00 UTC

Resolving

[Resolved] Fedora CI installability test affected by systemd/selinux issue.

Thu, 23 Nov 2023 00:00:00 +0000

Resolved 2024-02-09 10:00 UTC - The issue is resolved, please let us know if you would have still issues.

Update 2023-11-28 19:00 UTC - The problem still persists. Please follow this bugzzila https://bugzilla.redhat.com/show_bug.cgi?id=2250930 for more details.

Outage 2023-11-23 00:00 UTC - A systemd/selinux issue is causing unexpected failures in installability for a lot of components. Also some of the functional tests might be affected. For more info see https://pagure.io/fedora-ci/general/issue/447