Public Ranch on Testing Farm Status

[Resolved] Large queue on public ranch

Wed, 22 Apr 2026 10:17:00 +0000

Outage | 2026-04-22 10:17:00 UTC

The queue on public ranch has exploded, we are trying to scale up.

Update | 2026-04-22 23:44:00 UTC

Queues are back to normal. We were able to scale up, so we should have a bit more firepower here.

Resolved | 2026-04-22 23:45:00 UTC

Resolving

[Resolved] Elevated API errors

Mon, 20 Apr 2026 15:19:00 +0000

Outage | 2026-04-20 15:19:00 UTC

We are seeing elevated API errors, we are investigating the issue.

Update | 2026-04-20 16:14:00 UTC

This should be better now.

Resolved | 2026-04-20 17:52:00 UTC

Resolving, the issue is gone

[Resolved] Errors when regenerating tokens via the UI

Thu, 16 Apr 2026 11:37:00 +0000

Outage | 2026-04-16 11:37:00 UTC

We are aware of an issue when regenerating tokens, we are looking into the issue.

Resolved | 2026-04-20 16:15:00 UTC

This issue should be resolved for some time now

[Resolved] Fedora Rawhide and ELN composes provisioning errors out

Wed, 08 Apr 2026 14:53:00 +0000

Outage | 2026-04-08 14:53:00 UTC

Testing-farm jobs using the Fedora-ELN and Rawhide composes fail to provision or fail during the excution of the pre-artifact-installation playbook.

As far as we can tell it goes through something like: image booted, testing farm connected to it, started running playbooks, got to the tasks which upgrades all the rpms and reboots, could not connect back due to emergency mode.

Update | 2026-04-17 01:45:00 UTC

Also hitting Fedora-Rawhide now. Seems like some bug from Rawhide, investigation is ongoing.

Resolved | 2026-05-08 02:54:00 UTC

This is resolved via workarounds. Fedora-eln is now not updated to latest bits.

[Resolved] Testing Farm container based testing is broken

Wed, 25 Mar 2026 15:00:00 +0000

Outage | 2026-03-25 15:00:00 UTC

We are sorry, the rotation of secrets broken our quay.io creds.

Resolved | 2026-03-26 08:57:00 UTC

Resolved. Testing Farm container testing is back.

[Resolved] Testing Farm 2026-03.2 being deployed

Thu, 19 Mar 2026 19:18:00 +0000

Notice | 2026-03-19 19:18:00 UTC

We are deploying new Testing Farm version with tmt-1.69, stay tuned. Small downtime of API is expected, otherwise no other issues should arise. We will update this status entry if needed. ETA 4 hours.

Update | 2026-03-20 16:02:00 UTC

There was a small bug failing ~20 jobs in RH ranch causing Ansible to fail with: The conditional check ‘ansible_pkg_mgr in [‘unknown’, ‘dnf’]’ failed

Update | 2026-03-21 22:40:00 UTC

Testing Farm is now deployed, we will send out the release notes on Monday.

Update | 2026-03-21 22:42:00 UTC

There was an issue in Ansible setup for Fedora in Red Hat ranch, some of the jobs have failed, please restart them if you encountered Could not import the libdnf5 python module using /var/opt/standalone-python/bin/python3.

Resolved | 2026-03-23 19:02:00 UTC

Testing Farm was deployed! Enjoy!

[Resolved] Koji outage affects public ranch

Thu, 12 Mar 2026 11:13:00 +0000

Outage | 2026-03-12 11:13:00 UTC

Koji has an outage which affects test reqeuest with Koji or Copr artifacts. The affected requests end up with Test environment installation failed: Install packages error.

Resolved | 2026-03-12 15:39:00 UTC

Seems the koji is back and running. The incident is resolved.

[Resolved] Testing Farm outage - Cannot execute the rule

Thu, 05 Mar 2026 21:00:00 +0000

Outage | 2026-03-05 21:00:00 UTC

Both ranches are suffering outage after the most recent release deployment. A mistake in configuration is causing pipelines to fail. We are currently investigating why the corresponding fix have not been deployed properly.

Update | 2026-03-06 10:33:00 UTC

It was affecting Public Ranch too. Please restart your failed requests. The issue was fixed by reverting change. The issue was not affecting requests testing against containers.

Resolved | 2026-03-06 10:00:00 UTC

Hotfix deployed and new request are working.

[Resolved] Testing Farm 2026-02.2 was deployed on 2026-02-19

Thu, 19 Feb 2026 18:58:00 +0000

Outage | 2026-02-19 18:58:00 UTC

New Testing Farm version was deployed, but we will send out release notes on Monday, as we did not make it. For a previous of release notes see: https://testing-farm.gitlab.io/-/docs/root/-/jobs/13185451238/artifacts/preview/Testing%20Farm/0.1/releases/2026-02.2.html

Resolved | 2026-02-23 10:21:00 UTC

Resolving, new release is deployed fully.

[Resolved] Testing-farm jobs on rawhide fail signature check

Tue, 17 Feb 2026 12:14:00 +0000

Outage | 2026-02-17 12:14:00 UTC

Testing-farm jobs that require artifacts (for example scratch builds) are now failing on rawhide due to the signature enforcement proposal. We are investigating if this is an issue on our side or the change proposal.

For more info, follow https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BYG3JSPRB3R2YQWP6O3ZD2GMNFRVM7JS/#R2422WIXVQYIBCAXHHTWG7ITI2P6PERA

[Resolved] Large queue on public ranch due to provisioning outage

Mon, 16 Feb 2026 09:41:00 +0000

Outage | 2026-02-16 09:41:00 UTC

We just recovered from a provisioning outage on public ranch. Things should start resuming, but because if limited scalability it might take a while (1-2 hours)

Resolved | 2026-02-16 11:11:00 UTC

The queue is gone, outage resolved.

[Resolved] Some requests in Public Ranch are missing artifacts

Wed, 11 Feb 2026 00:00:00 +0000

Outage | 2026-02-11 00:00:00 UTC

We are occasionally experiencing an issue with artifacts archivation. Some requests in Public Ranch have incomplete results.

Issue is tracked in: https://issues.redhat.com/browse/TFT-4328

Resolved | 2026-03-23 20:51:00 UTC

This issue should be now resolved with 2026-03.2 release.

[Resolved] Broken Fedora repositories

Tue, 10 Feb 2026 23:00:00 +0000

Outage | 2026-02-11 00:00:00 UTC

We are experiencing outage of repositories on Fedora images.

Issue is tracked in: https://forge.fedoraproject.org/infra/tickets/issues/13138

Updating and loading repositories:
 Fedora 43 openh264 (From Cisco) - x86_ 100% |   8.6 KiB/s |  10.8 KiB |  00m01s
Failed to download metadata
 Error loading local metadata for repository "updates"
  Librepo error: Checksum error /var/cache/libdnf5/updates-431245df5652aba0/repodata/72324174538178d41d9e8bc090d9eb64666f071a34c516937f13e6932546a129-filelists.xml.zck: Unable to read zchunk lead

Resolved | 2026-02-17 01:02:00 UTC

Resolving, seems the issue was addressed and we do not see it anymore.

[Resolved] Fedora 44 not available in Testing Farm

Mon, 09 Feb 2026 11:46:00 +0000

Outage | 2026-02-09 11:46:00 UTC

The import is in progress after branching. We are working it out. ETA later today.

Resolved | 2026-02-10 02:45:00 UTC

Fedora 44 is now available.

[Resolved] High queue on public ranch, expect delays

Thu, 05 Feb 2026 11:00:00 +0000

Outage | 2026-02-05 11:00:00 UTC

We are experiencing high load on Testing Farm. Investigation in place. Expect delays.

Update | 2026-02-05 15:00:00 UTC

The situation is ongoing, current average queue time is 3.5 hours

Update | 2026-02-06 01:40:00 UTC

Things are improving, we had to kill testing from scap via atex. Seems like some new integration using reservations is not cleaning up resources, causing all the requests to stay up long and take all resources, blocking other users.

Update | 2026-02-06 02:00:00 UTC

queue is almost gone, we are monitoring the situation

Resolved | 2026-02-09 10:26:00 UTC

Problems should be gone as of last week. The misusage was addressed on the user side.

[Resolved] Testing Farm 2026-02.1 deployment in progress

Mon, 02 Feb 2026 10:46:00 +0000

Outage | 2026-02-02 10:46:00 UTC

We are deploying a new TF release, brief outages of API might happen during this time.

Update | 2026-02-02 18:51:00 UTC

Deployment is half done, and ongoing to the next day

Resolved | 2026-02-04 15:59:00 UTC

Testing Farm 2026-02.1 is fully deployed. Resolving.

[Resolved] High load - delays and timeouts

Thu, 22 Jan 2026 00:00:00 +0000

Outage | 2026-01-22 00:00:00 UTC

We are working to cancel the problematic jobs. We apologize for the inconvenience.

Update | 2026-01-22 20:43:00 UTC

Things should be finally resuming. We were today hamerred by requests trying to install 3k+ artifacts from ELN. Unfortunately the initial clenaup did not go well and we found a bug preventing the jobs to die. That caused delays. We are sorry for the trouble. The queue should start slowly go down. We are trying to scale up to deal with the current queue.

Resolved | 2026-01-23 00:00:00 UTC

All problematic jobs removed and queue is back to 0. Thank you for you patience.

[Resolved] Elevated API errors in Testing Farm

Mon, 19 Jan 2026 09:00:00 +0000

Outage | 2026-01-19 09:00:00 UTC

We are experiecing read timeouts on our API. Issue is under investigation.

Update | 2026-01-19 20:00:00 UTC

We are patching API to better recycle stale connections

Update | 2026-01-19 20:21:00 UTC

Issue should be addressed, we are monitoring the situation. https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 20:34:00 UTC

Things are back to stable, we hotpatched production with: https://gitlab.com/testing-farm/nucleus/-/merge_requests/266

Update | 2026-01-19 21:11:00 UTC

The issues is back :( we are looking further

Resolved | 2026-01-19 23:33:00 UTC

Resolving, we did nto see this anymore after applying the fixes.

[Resolved] Redoc documentation shows empty page

Thu, 08 Jan 2026 14:03:00 +0000

Outage | 2026-01-08 14:03:00 UTC

We have a dependencies problem which affects https://api.testing-farm.io/redoc page rendering it empty. Please use https://api.testing-farm.io/docs instead for now. We are working on a fix

Resolved | 2026-01-15 19:03:00 UTC

resolved with 2026-01.1 release

[Resolved] Public ranch outage

Tue, 16 Dec 2025 13:00:00 +0000

Outage | 2025-12-16 13:00:00 UTC

Unfortunately the provisioner went down, and it took a while we noticed :(

Update | 2025-12-17 03:40:00 UTC

The outage was addressed, and the provisioner is getting back into good shape, expect failed jobs and delays though.

Update | 2025-12-17 03:42:00 UTC

There is around 1.7k requests to process, will take few hours

Update | 2025-12-17 20:10:00 UTC

Things are back to normal from the morning. Resolving.

Update | 2025-12-19 03:30:00 UTC

Outage should be gone.

Resolved | 2025-12-19 03:30:00 UTC

Resolving

[Resolved] Public ranch requests testing against copr builds are broken

Wed, 10 Dec 2025 15:30:00 +0000

Outage | 2025-12-10 15:30:00 UTC

Copr added anubis that broke copr builds installation in Testing Farm. We are trying to get the team to revert it until we can find out how to solve the issue.

Update | 2025-12-10 20:06:00 UTC

Things look resolved. Copr team applied a fix to not put repo files under annubis.

Resolved | 2025-12-10 20:08:00 UTC

Resolved.

[Resolved] Public Ranch outage - jobs not finishing, high queue

Fri, 28 Nov 2025 02:00:00 +0000

Outage | 2025-11-28 02:00:00 UTC

Hi, Testing Farm got overloaded in the early morning and out provisioner did not make it. We are trying to get it back to a stable state. Expect jobs that do not finish or error out and new requests being queued for long time. We appologize for the inconvenience.

Update | 2025-11-29 00:53:00 UTC

Resolving, things should be back to stable now.

Resolved | 2025-12-01 08:27:00 UTC

Resolved for some time, closing.

[Resolved] Public Ranch provisioning unstable, high queue

Wed, 12 Nov 2025 13:00:00 +0000

Outage | 2025-11-12 08:00:00 UTC

We are seeing issues with provisioning in Public ranch. Investigation is in progress.

Resolved | 2025-11-16 20:47:00 UTC

Resolving, after updating to Testing Farm v0.0.82 with a fix seems provisioning is stable.

[Resolved] quay.io is down

Mon, 20 Oct 2025 07:51:00 +0000

Outage | 2025-10-20 07:51:00 UTC

We are aware of quay.io outage, we are gatheting more information. This will affect our services, but the real extent is not yet known.

Update | 2025-10-20 07:58:00 UTC

Seems connected to the AWS ouatage in us-east-1: https://health.aws.amazon.com/health/status

Update | 2025-10-20 09:10:00 UTC

Outage was resolved by routing traffic to us-east-2.

Resolved | 2025-10-20 09:10:00 UTC

Resolving

[Resolved] Provisioning unstable in Testing Farm Public Ranch

Sat, 18 Oct 2025 10:00:00 +0000

Outage | 2025-10-18 10:00:00 UTC

During this time the outage started, the operation team was out on weekend.

Update | 2025-10-19 15:00:00 UTC

The outage was resolved by a Testing Farm engineering checking in during the weekend.

Update | 2025-10-20 08:06:00 UTC

Provisioning looks stable now, but keeping open for awareness. We are tracking the issue in https://issues.redhat.com/browse/TFT-3977.

Resolved | 2025-11-06 20:51:00 UTC

Resolving, we are not seeing any issues here.

[Resolved] Testing Farm 2025-09.1 was deployed

Fri, 17 Oct 2025 11:23:00 +0000

Outage | 2025-10-17 11:23:00 UTC

We are preparing release notes.

Update | 2025-11-12 12:41:00 UTC

Release notes are available:

https://docs.testing-farm.io/Testing%20Farm/0.1/releases/2025-09.1.html

Resolved | 2025-11-12 12:43:00 UTC

Resolving

[Resolved] Fedora Koji Outage

Wed, 15 Oct 2025 07:30:00 +0000

Outage | 2025-10-15 07:30:00 UTC

The Fedora Koji is down. No ETA available currently.

We are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Update | 2025-10-16 00:06:00 UTC

The experience is better, but still elevated 503. Follow this issue to find out more: https://pagure.io/fedora-infrastructure/issue/12814

Resolved | 2025-11-12 12:52:00 UTC

Seems the issues are gone according to latest updates of https://pagure.io/fedora-infrastructure/issue/12814

[Resolved] Fedora Koji Outage

Tue, 14 Oct 2025 10:41:00 +0000

Outage | 2025-10-14 10:41:00 UTC

The Fedora Koji is down. No ETA available currently.

Update | 2025-10-14 10:43:00 UTC

We are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Resolved | 2025-10-14 11:28:00 UTC

We didn’t get any statement, but Koji started to work again.

[Resolved] Provisioning errors in Public Ranch due to an outage

Mon, 01 Sep 2025 16:00:00 +0000

Outage | 2025-09-01 16:00:00 UTC

The newly created public workers did not have access to Artemis causing provisioning errors during 18:00 UTC to ~ midnight.

Update | 2025-09-02 00:53:00 UTC

The problem was addressed and things should be getting back to normal. Please restart your failed tests.

Resolved | 2025-09-02 17:27:00 UTC

Resolved

[Resolved] Testing Farm public ranch large queue

Mon, 01 Sep 2025 03:00:00 +0000

Outage | 2025-09-01 03:00:00 UTC

We are aware of a large queue. We are investigating the issue. Sorry for the delay :(

Resolved | 2025-09-01 16:34:00 UTC

The outage was resolved by scaling up the deployment. The large queue although is a bit mystery. Anyway, I did not find any direct indication of a problem.

[Resolved] Fedora provisioning errors

Tue, 19 Aug 2025 17:23:00 +0000

Outage | 2025-08-19 17:23:00 UTC

We are seeing elevated provisioning issues on Fedora Rawhide mostly. The issues is under investigation.

Update | 2025-08-26 17:53:00 UTC

The issue was found, it is caused by a bug in cloud-init causing wrong ordering of services at startup, breaking in some ocassions sshd. See https://bugzilla.redhat.com/show_bug.cgi?id=2390898 for details. We are waiting for Fedora-Rawhide and Fedora-43 fixed builds to land. Hopefully that will stabilize the testing.

Resolved | 2025-09-01 11:49:00 UTC

This is finally resolved. Both Rawhide and F43 should be stable now.

[Resolved] Public Ranch outage

Tue, 05 Aug 2025 20:00:00 +0000

Outage | 2025-08-05 20:00:00 UTC

Due to a broken patch the worker images do not start causing the requests to stay in queued state. We are working out to resolve the issue. It might take some time to process all the requests.

Update | 2025-08-06 09:57:00 UTC

The outage is resolved, we are restarting queued up jobs in batches, can take some time to process. Feel free to restart your testing if you are in hurry. We will resolve the outage once we processed the whole queue.

Update | 2025-08-06 21:00:00 UTC

We restarted all the remaining jobs, expect larger queue in the next hours.

Update | 2025-08-18 08:37:00 UTC

Closing, outage resovled as previously announced.

[Resolved] Fedora Infra outage affecting koji and dist-git

Mon, 04 Aug 2025 06:00:00 +0000

Outage | 2025-08-04 06:00:00 UTC

We are aware of outage of Fedora Infrastructure. The testing will be blocked or delayed. No ETA yet, we are watching fedora infrastructure matrix channel for details: https://matrix.to/#/#admin:fedoraproject.org

Resolved | 2025-08-05 08:29:00 UTC

The outage is resolved, Fedora infra should be back.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Thu, 31 Jul 2025 07:00:00 +0000

Outage 2025-07-31 07:00:00 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits.

Resolved 2025-07-31 21:00:00 UTC - The outage s resolved. Please restart your test request if they were affected.

[Resolved] Public Ranch deployed to 2025-06.1 release

Mon, 14 Jul 2025 21:00:00 +0000

Outage | 2025-07-14 21:00:00 UTC

We deployed a new Testing Farm release. If you see some issues let us know please.

Resolved | 2025-07-17 01:07:00 UTC

Resolved

[Resolved] Storage almost full - purging

Wed, 09 Jul 2025 07:01:00 +0000

Outage | 2025-07-09 07:01:00 UTC

Disk space is almost full. Old very large artifacts will be purged.

Resolved | 2025-07-09 09:17:00 UTC

Available archive space increased.

[Resolved] src.fedoraproject.org is down

Thu, 26 Jun 2025 10:00:00 +0000

Outage | 2025-06-26 10:00:00 UTC

DC reloctation outage expected. Pleases see status here https://status.fedoraproject.org/ CentOS pkg repos are also affected.

Resolved | 2025-07-14 07:53:00 UTC

The outage is resolved. All Fedora systems are back to normal. Resolving outage.

[Resolved] IT resilience testing multiple SP services down

Fri, 13 Jun 2025 13:00:00 +0000

Outage | 2025-06-13 13:00:00 UTC

tps-server

Resolved | 2025-06-13 21:00:00 UTC

Testing complete

[Resolved] Fedora Rawhide broken due to Python3.14 upgrade

Fri, 13 Jun 2025 10:21:00 +0000

Outage | 2025-06-13 10:21:00 UTC

Unfortunately Fedora Rawhide testing is broken due to Python3.14 which breaks test environment preparation, we are looking how to resolve this with Fedora development, but do not expect it to work today. Sorry for the inconvenience.

Resolved | 2025-06-30 09:14:00 UTC

We believe the testing is now working well in Rawhide. Note that there might be still issue with components broken due to Python3.14 update, but in general TF should be able to provision and run tests against Fedora Rawhide.

[Resolved] Public ranch testing blocked

Thu, 12 Jun 2025 20:00:00 +0000

Outage | 2025-06-12 20:00:00 UTC

The deployment of some workers failed and they cannot access our provisioner Artemis. All jobs provisioning VMs are blocked. We are looking how to reoslve the problem.

Resolved | 2025-06-13 10:20:00 UTC

The outage was resolved.

[Resolved] Public ranch provisioning is blocked

Mon, 26 May 2025 14:00:00 +0000

Outage | 2025-05-26 14:00:00 UTC

The requests will be stuck in running state until resolved.

Update | 2025-05-26 18:31:00 UTC

The issue was indentified, but we are not able to yet find a workaround. No ETA currently available.

Update | 2025-05-26 19:11:00 UTC

The outage was resolved. Currently we do not see any effects of the outage. If you run into provisioning errors, please restart your requests.

Resolved | 2025-05-27 06:58:00 UTC

Outage was resolved.

[Resolved] ansible-core breaks ansible-lint

Tue, 20 May 2025 07:42:00 +0000

Outage | 2025-05-20 07:42:00 UTC

Tooling affected by https://github.com/ansible/ansible-lint/issues/4611

Update | 2025-08-07 12:00:00 UTC

This has been mostly mitigated by forcing the ansible version to something before 2.19.

Update | 2025-08-18 09:12:00 UTC

Seems resolved for some time now.

Resolved | 2025-08-19 17:20:00 UTC

Resolving

[Resolved] Gitlab.com outage causing various requests to fail

Mon, 19 May 2025 21:00:00 +0000

Outage | 2025-05-19 21:00:00 UTC

Gitlab.com has a wide outage, see https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/682bb3b0158484053205f59b. If the requests use gitlab.com, they might randomly fail on 500 and similar.

Resolved | 2025-05-20 18:00:00 UTC

Things look more stable now. Resolving.

[Resolved] rpminspect requests failing on out-of-memory

Sat, 17 May 2025 18:00:00 +0000

Outage | 2025-05-17 18:00:00 UTC

Due to a deployment error the rollback of a change did not happen, and some rpminspect jobs could be terminated on out-of-memory.

Update | 2025-05-18 23:51:00 UTC

The deployment was fixed, the issue should not appear anymore.

Update | 2025-05-18 23:00:00 UTC

Issue was addressed by rolling back the jobs change.

Resolved | 2025-05-18 23:00:00 UTC

Outage resolved.

[Resolved] Rawhide requests erroring out on Testing Farm

Mon, 12 May 2025 12:00:00 +0000

Outage | 2025-05-12 12:00:00 UTC

We are experiencing high error rate on Public ranch.

Update | 2025-05-12 20:31:00 UTC

Unfortunately the Rawhide composes have expired, we did not notice the update job failing :(

Resolved | 2025-05-12 21:18:00 UTC

The outage was resolved. Please restart any jobs/requests against Rawhide.

NVIDIA V100 GPU not available

Mon, 05 May 2025 06:00:00 +0000

Outage | 2025-05-05 06:00:00 UTC

Unfortunately the instances with NVIDIA V100 GPUs are not available in the availability zone we use. AWS is working on scaling up to meet the demand, but there is currently no ETA. Use the NVIDIA K80 instances if you can, if not please wait, we are working on moving our workloads to multiple AZs, but it might take a few days :( Sorry for the inconvenience.

[Resolved] Some requests cancelled by an error.

Mon, 14 Apr 2025 09:52:00 +0000

Outage | 2025-04-14 09:52:00 UTC

Some of the requests might have canceled by admin error. Please restart your testing if your request was canceled unexpected. We are sorry for the inconvenience.

Resolved | 2025-04-14 10:00:00 UTC

Resolving this notice after 3 days.

[Resolved] Testing Farm outage due to deployment

Thu, 03 Apr 2025 07:30:00 +0000

Resolved 2025-04-03 9:00 UTC - Testing Farm 2025-01.4 is released. Queue getting crunched

Outage 2025-04-03 7:30 UTC - Both ranches are disabled due to messed up deployment. Expect long queue times.

[Resolved] Public Ranch Outage - outdated composes

Tue, 01 Apr 2025 14:00:00 +0000

Resolved 2025-04-01 14:47 UTC - The outage is resolved. Outage 2025-04-01 14:00 UTC - Public Ranch provisioning is disrupted due to outdated images, fix is landing in 20 minutes.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Thu, 06 Mar 2025 15:30:00 +0000

Resolved 2025-03-06 15:30 UTC - This problem is not yet addressed completely. But we are taking care of it manually.

Outage 2025-04-03 7:30 UTC - Both ranches are disabled due to messed up deployment. Expect long queue times.

Update 2025-03-06 22:00 UTC - Provisioning unblocked by deleting some security groups, we will monitor this.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Fri, 07 Feb 2025 14:00:00 +0000

Resolved 2025-02-07 16:00 UTC - Issue was addressed, but the root cause - a bug in the provisioner is not yet fixed. We hope to fix it in next release.

Outage 2025-02-07 14:00 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits.

[Resolved] Public Ranch Outage - expired composes

Sun, 02 Feb 2025 04:00:00 +0000

Resolved 2025-02-03 08:00 UTC - Images had been refreshed. Please restart your testing, all jobs affected will error out eventually.

Outage 2025-02-02 04:00 UTC - Public ranch composes have expired, we are working on refreshing them. Sorry for the inconvenience.

[Resolved] Public Ranch Outage - artifacts storage out of space

Wed, 29 Jan 2025 16:00:00 +0000

Outage 18:00 UTC - Resolve outage, it was resolved in ~ 2 hours.

Outage 16:00 UTC - Artifacts storage on public storage went full. We are working on the issue.

[Resolved] Public Ranch Outage - bug in security groups cleanup.

Thu, 16 Jan 2025 15:20:00 +0000

Resolved 2025-01-16 16:10 UTC - The outage s resolved. Please restart your test request if they were affected.

Outage 2025-01-16 15:20 UTC - Public Ranch provisioning is disrupted due to hitting the security groups limits. ETA is 30 minutes.

[Resolved] Fedora Rawhide outdated image

Tue, 14 Jan 2025 08:00:00 +0000

Resolved 2025-01-15 10:00 UTC - New configuration deployed, fixing Rawhide provisioning in public ranch.

Outage 2025-01-14 08:00 UTC - Rawhide image has been deleted and automation for updating looks broken, investigating.

[Resolved] Fedora Rawhide outdated image

Thu, 09 Jan 2025 08:00:00 +0000

Outage 2025-01-09 12:00 UTC - Image updated, outage is resolved.

Outage 2025-01-09 08:00 UTC - Rawhide image has been deleted and new one updated yesterday is alredy missing too, investigating.

[Resolved] Fedora 41 and Rawhide outage on public ranch - 0xe2 breaking ansible

Fri, 15 Nov 2024 00:00:00 +0000

Resolved 2024-11-18 23:15 UTC - Fix for the issue had been deployed and Fedora-41 and Rawhide should work again.

Update 2024-11-15 13:00 UTC - dnf bug triggering our bug: https://github.com/rpm-software-management/dnf5/issues/1865

Update 2024-11-15 11:20 UTC - fix and test is ready, working on deployment

Update 2024-11-15 11:00 UTC - issue is in ensure_str(), inside gluetool. Fixing now and working on extending test

Outage 2024-11-15 00:00 UTC - Fedora 41 and rawhide affected by ansible crashed on decoding UTF-8 char 0xe2

[Resolved] Misconfigured worker in public ranch

Fri, 08 Nov 2024 08:00:00 +0000

Resolved 2024-11-08 15:00 UTC - Outage was resolved, out workers lost connection to Artemis, manual cleanup of SG resolved the issue.

Outage 2024-11-08 08:00 UTC - One of the public workers was misconfigured, causing connection problems for a small number of test number of test requests.

[Resolved] Testing Farm 2024-10.1 released

Thu, 07 Nov 2024 08:00:00 +0000

Resolved 2024-11-11 08:00 UTC - Everything looks ok, end of outage

Update 2024-11-11 09:00 UTC - Release should be stable, the other outages are not related to the release itself. Anyway, keeping this here as a note for one more week to raise awareness in case somebody runs into issues.

Update 2024-11-07 18:30 UTC - We should be stable now and deployed to 2024-10.1. There can be an issue where the results viewer is not loading some older results, no data gone, just it does nto present the data and you get an empty screen, We are trying to resolve it.

Update 2024-11-07 09:30 UTC - A bit of queue can happen. Please be patient.

Update 2024-11-07 09:00 UTC - Note that the new release is introducing some breaking changes, see a preview in the https://gitlab.com/testing-farm/docs/root/-/merge_requests/153[release notes draft].

Update 2024-11-07 08:10 UTC - A short outage caused some jobs to error out on Public Ranch, please restart your jobs.

Info 2024-11-07 08:00 UTC - We are deploying Testing Farm 2024-10.1. We are trying to avoid outages, but if you see something, check this status.

[Resolved] Testing Farm API down

Wed, 06 Nov 2024 14:30:00 +0000

Resolved 2024-11-06 15:10 UTC - The outage is over. API is back. Please restart your jobs if you were affected. We are sorry for the inconvenience.

Outage 2024-11-06 14:30 UTC - We are suffering from an API outage, jobs will break, can be stuck forever.

[Resolved] Fedora 41 outage on public ranch.

Fri, 01 Nov 2024 00:00:00 +0000

Resolved 2024-11-01 15:00 UTC - Fedora 41 images are updated and working now.

Outage 2024-11-01 00:00 UTC - Fedora 41 images are gone due to a rename, we are working on resolving the issue.

[Resolved] Public Ranch Outage - no archive disk space

Tue, 01 Oct 2024 10:00:00 +0000

Resolved 11:00 UTC - Alert created for archive disk space. Purged old files.

Outage 10:00 UTC - Public Artemis stopped working, investigating. Until this is resolved, request will stay queued and will be processed later.

[Resolved] Public Ranch - broken Fedora repositories

Mon, 16 Sep 2024 00:00:00 +0000

Resolved 2024-09-16 19:00 UTC - The outage was resolved by the Fedora infrastructure team.

Outage 2024-09-16 00:00 UTC - We are experiencing outages due to broken repositories in Fedora. The issue is tracked by Fedora Infrastructure in this issue - https://pagure.io/fedora-infrastructure/issue/12183. The issue causes following failures in Testing Farm:

Updating and loading repositories:                                                            
 Fedora 41 openh264 (From Cisco) - x86_64             100% |   3.0 KiB/s |   6.0 KiB |  00m02s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 385.2 KiB | -00m00s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 385.2 KiB | -00m00s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 385.2 KiB | -00m00s
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 391.1 KiB | -00m00s
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
 Fedora 41 - x86_64              100% [==================] |  12.3 KiB/s | 391.1 KiB | -00m00s
...snip...
>>> Downloading successful, but checksum doesn't match. Calculated: f6dc87bb2fab2e324c8c8f3980
>>> Downloading successful, but checksum doesn't match. Calculated: f6dc87bb2fab2e324c8c8f3980
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055
>>> Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirror
Failed to download metadata (metalink: "https://mirrors.fedoraproject.org/metalink?repo=fedora-41&arch=x86_64") for repository "fedora"
 Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[Resolved] Public ranch outage - workers not able to connect to Artemis.

Wed, 28 Aug 2024 10:29:00 +0000

Resolved 2024-08-28 14:21 UTC - the service is fully operational. If your requests are still failing, please restart them.

Outage 2024-08-28 10:29 UTC - because of misconfigured network in production environment, workers are not able to connect to Artemis.

[Resolved] Fedora 40 small disk size

Tue, 20 Aug 2024 18:00:00 +0000

Resolved 2024-08-29 08:00:00 UTC - Resolved by updating to new, not broken F40 image

Degradation 2024-08-20 18:00 UTC - Due to recent btrfs changes in Fedora 40, the auto-resizing of root partition to the cloud image is not working. For details see https://bodhi.fedoraproject.org/updates/FEDORA-2024-67f6df918c and https://bodhi.fedoraproject.org/updates/FEDORA-2024-d44bd4abd9 bodhi updates. The issue should be fixed in 2024-08-22 once these fixes get into the fresh compose.

[Resolved] Testing Farm API outage and longer queues on both ranches.

Thu, 08 Aug 2024 04:00:00 +0000

Outage 2024-08-08 06:15 UTC - We resumed normal operation. Outage is over.

Outage 2024-08-08 04:00 UTC - We experienced an api outage. If you encounted some issues, please restart your requests.

[Resolved] All Fedora images outdated, public ranch requests processing paused.

Mon, 05 Aug 2024 04:00:00 +0000

Resolved 2024-08-05 13:00 UTC - The outage was resolved by uploading images to Fedora AWS ourselves. We are still waiting for a proper fix on fedora infrasturcture side.

Update 2024-08-05 12:30 UTC - The upload is progressing, we had to implement few features to make the uploaded images look same as with the original ones, e.e. bios mode unset, gp3 used. ETA 1h.

Update 2024-08-05 09:30 UTC - The uploading of the images is broken, no new updates for long time :(. See https://pagure.io/fedora-infrastructure/issue/12110 for details. We stopped processing public ranch queue until the outage is resolved, we are uploading the images manually.

Outage 2024-08-05 08:00 UTC - All Fedora images have expired, we are investigating the issue.

[Resolved] Fedora Rawhide outdated image

Fri, 02 Aug 2024 08:00:00 +0000

Update 2024-08-02 09:00 UTC - New configuration deployed, fixing Rawhide provisioning in public ranch

Outage 2024-08-02 08:00 UTC - Rawhide image has been deleted and automation for updating looks broken, investigating.

[Resolved] Public and Redhat Ranch Outage - broken deployment

Thu, 25 Jul 2024 13:00:00 +0000

Update 19:45 UTC - almost all jobs from queue has been processed.

Update 18:18 UTC - deployemnt fixed, queues are being processed but it will take some time.

Outage 13:00 UTC - Issue with deployment affecting all new jobs.

[Resolved] Public Ranch Outage - artifacts out of space

Tue, 23 Jul 2024 06:00:00 +0000

Resolved 10:00 UTC - The issue was resolved. Resolving outage.

Update 09:00 UTC - We started to investigate the issue and working on getting back TF public ranch into a good state.

Outage 06:00 UTC - Testing Farm Artifacts are out of space.

[Resolved] Public Ranch Outage - complete outage, request will stay queued or error out

Mon, 22 Jul 2024 00:16:00 +0000

Resolved 09:17 UTC - The issue was resolved, our postgres database went out of space in the early morning. All affected requests restarted. Fedora CI users of Testing Farm, will need to manually restart their testing. Packit users pipelines should automatically resolve.

Outage 07:17 UTC - Public Artemis stopped working, investigating. Until this is resolved, request will stay queued and will be processed later.

[Resolved] Public Ranch Outage - possible large queue due to deadlocked workers

Wed, 15 May 2024 20:00:00 +0000

Resolved 2024-05-16 19:30 UTC - problems should be gone

Monitoring 2024-05-16 07:30 UTC - the queue should be gone, we are investigating the root cause

Outage 2024-05-16 01:00 UTC - we started to investigate the issue

Outage 2024-05-15 20:00 UTC - large queue started to appear

[Resolved] Public Ranch Outage - artifacts storage out of space

Thu, 09 May 2024 16:00:00 +0000

Update 19:30 UTC - Old requests with large space usage were removed. Outage resolved.

Outage 16:00 UTC - Artifacts storage on public storage went full.

[Resolved] Public Ranch Outage - F40 images expired

Mon, 06 May 2024 00:00:00 +0000

Update 7:30 UTC - Issue was addressed. F40 image was updated.

Outage 00:00 UTC - F40 image got expired.

[Resolved] Public Ranch Outage - large queue, high error rate due to broken workers

Tue, 30 Apr 2024 18:00:00 +0000

Update 21:30 UTC - Outage was resolved.

Update 20:30 UTC - Issue was addressed. Queue is being processed. Some jobs might have failed, please restart your jobs if they errored out.

Outage 18:00 UTC - We broke our workers and they are not able to connect to our provisioner. We are investigating the issue.

[Resolved] Public Ranch Outage - Broken Fedora Rawhide images

Thu, 25 Apr 2024 00:00:00 +0000

Resolved 11:00 UTC Workaround with installing python3-dnf deployed

Outage 00:00 UTC - Our images in public ranch have broken dnf installation, missing python3-dnf

[Resolved] Public Ranch Outage - Random issues with access to koji

Thu, 25 Apr 2024 00:00:00 +0000

Outage 2024-05-06 7:15 UTC - The issue looks resolved, we did not see it for some time.

Outage 2024-04-25 00:00 UTC - Random connection issues (timeout) for kojihub.stream.centos.org affecting CentOS-Stream images

[Resolved] Public Ranch Outage - Expired images

Mon, 15 Apr 2024 00:00:00 +0000

Resolved 08:30 UTC - Resolved. Images have been updated.

Outage 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Public Ranch Outage - Fedora 40 expired

Fri, 29 Mar 2024 00:00:00 +0000

Resolved 17:00 UTC - Our F40 image has expired.

Outage 00:00 UTC - Our F40 image has expired.

[Resolved] Public Ranch Outage - Expired images

Thu, 28 Mar 2024 00:00:00 +0000

Resolved 09:30 UTC - The images were bumped, sorry for the outage.

Outage 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Public Ranch Outage - Artemis requests in error state

Tue, 19 Mar 2024 00:00:00 +0000

Outage 11:00 UTC - Images were updated. We are adding finally automation to get them updated regularly.

Outage 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Long queues in Public and Red Hat ranch

Tue, 12 Mar 2024 15:00:00 +0000

Resolved 20:00 UTC (next day) - The problems should be gone. If you see weird failures please restart, unfortunately we had outage with artifacts storage, so rsync might have timed out. Should be all ok now.

Outage 15:00 UTC - Long queues detected, due to workaround gone bad. We are scaling and hopefully the issues should be over. Please restart if you run into some issues. Testing should be otherwise stable.

[Resolved] Public Ranch artifact storage ran out of disk space

Fri, 08 Mar 2024 08:00:00 +0000

Outage 8:00 UTC - Testing Farm public ranch artifact storage ran out of disk space. Some requests might end unexpectedly, feel free to restart them. We are working on cleaning up old large artifacts and increasing disk space.

[Resolved] Public Ranch large queue - expect delays

Thu, 07 Mar 2024 10:00:00 +0000

Resolved* 15:00 UTC - Testing Farm queue is large again due to an outage. Expect delays, ETA 1 hour.

Resolved* 12:00 UTC - Testing Farm is deployed. Queue is processed. Normal operation expected.

Outage 10:00 UTC - Testing Farm public ranch has long queue due to ongoing deployment. It might take a bit of time the queue to go down. Please be patient.

[Resolved] Public Ranch large queue - expect delays

Tue, 05 Mar 2024 15:00:00 +0000

Outage 21:00 UTC - Outage was resolved.

Outage 15:00 UTC - Testing Farm public ranch has long queue due to deployment. It might take a bit of time the queue to go down. Please be patient.

[Resolved] Public Ranch large queue - expect delays

Thu, 15 Feb 2024 17:00:00 +0000

Resolved 19:30 UTC - Queue is almost gone, resolving.

Outage 17:00 UTC - Testing Farm public ranch has long queue due to deployment. It might take a bit of time the queue to go down. Please be patient.

[Resolved] Public Ranch Outage - Artemis requests in error state

Sun, 10 Dec 2023 00:00:00 +0000

Resolved 2023-12-11 10:00 UTC - The images were updated. The issue should be resolved.

Outage 2023-12-10 00:00 UTC - Our images in public ranch have expired. We are updating them.

[Resolved] Public Ranch Outage

Sat, 02 Dec 2023 00:00:00 +0000

Outage 2023-12-02 03:00 UTC - Outage was resolved. Please restart your tests if your tests failed unexpectedly.

Outage 2023-12-02 00:00 UTC - We need to reprovision our Kubernetes cluster and we hit some issues trying to do that without outage. Public Ranch will be paused for 12 hours at least. Sorry for the inconvenience.

[Resolved] Public Ranch reservations are broken

Wed, 29 Nov 2023 00:00:00 +0000

Resolved 2023-12-07 10:00 UTC - Deployment has been changed to terragrunt, the issues is resolved, reservations should work according to the IP allowlist.

Info 2023-11-29 00:00 UTC - Due to deployment changes we broke the Testing Farm reservations in public, please bare with us until we resolve the issues. We hope to resolve the problems today.

[Resolved] Testing Farm Public Ranch outage due to expired images.

Mon, 20 Nov 2023 00:00:00 +0000

Resolved 10:00 UTC - Outage was resolved, images were updated.

Outage 00:00 UTC - Some of our public images have expired, we are working to get them updated.

[Resolved] Expired images on Public ranch.

Sat, 30 Sep 2023 04:00:00 +0000

Resolved 2023-10-02 04:00 UTC - Images were updated, problems should be gone. Restart of testing will be needed for affected users.

Outage 2023-09-30 4:00 UTC - Some of the images have expired on Public Ranch.

[Resolved] Console logs are disabled due to a discovered bug

Mon, 18 Sep 2023 14:00:00 +0000

Resolvede 2023-09-19 21:00 UTC

Issue was resolved in Artemis: https://gitlab.com/testing-farm/artemis/-/merge_requests/1160

Outage 2023-09-18 8:00 UTC

We have found a bug we need to address and had to temporarily disable the console log functionality. ETA 1-2 days.

[Resolved] Console log has wrong link in results viewer or not available at all.

Fri, 15 Sep 2023 08:00:00 +0000

Resolved 2023-09-26 18:00 UTC

Console log links are now correct with a hotfix. Resolving outage.

Outage 2023-09-15 8:00 UTC

Red Hat ranch: Console log has wrong link in the results viewer for Red Hat ranch. We are looking into the problem. Until then, please remove the work-.* path from the URL. For example http://artifacts.osci.redhat.com/testing-farm/10de2a43-a2a1-4c4f-ab57-cf5e4416c93b/work-create-osbuild1e5bpd1y/console-94d0f6af-27a9-416c-9412-32bcf440dcce.log should be http://artifacts.osci.redhat.com/testing-farm/10de2a43-a2a1-4c4f-ab 57-cf5e4416c93b/console-94d0f6af-27a9-416c-9412-32bcf440dcce.log, note the work-create-osbuild1e5bpd1y path removed in this case to get a valid link.

Public ranch: Console logs are not yet working on Public ranch, we are working to get them available also.

[Resolved] Public ranch testing blocked

Wed, 19 Jul 2023 08:00:00 +0000

Resolved | 2023-07-19 11:00:00 UTC

Problem was resolved by applying a hotfix to guest-setup playbooks.

Outage | 2023-07-19 08:00:00 UTC

We are trying to update the images for Rawhide, but it breaks due to dnf5 again. Testing is completely blocked until we resolve the problem. ETA 2 hours.

[Resolved] Fedora Rawhide testing blocked on dnf5 breakage

Tue, 18 Jul 2023 08:00:00 +0000

Update

Rawhide is working again \o/.

Outage

The switch to dnf5 broke tmt. We are upgrading to tmt 1.25. We hope to resolve the problem today.

[Resolved] Fedora Rawhide testing blocked on dnf5

Thu, 22 Jun 2023 09:00:00 +0000

Resolved - All issues were addressed, Fedora-Rawhide testing works again \o/

Outage - The switch to dnf5 broke Testing Farm again. Some use cases might work though, no current ETA to resolve all the problems. We will be posting updates here.

[Resolved] Fedora Rawhide testing blocked on dnf5

Wed, 21 Jun 2023 12:30:00 +0000

Resolved - All issues were addressed, Fedora-Rawhide testing works again \o/

Outage - The switch to dnf5 broke Testing Farm. We are trying to hotpatch the problems to bring back testing against Rawhide.

[Resolved] Expired Fedora-Rawhide images in Public Ranch

Mon, 29 May 2023 02:00:00 +0000

Resolved - Outage was resolved, failed jobs will require restart.

Outage - Images for Fedora Rawhide have expired. We are refreshing it. ETA 30 min.

[Resolved] Unexpected outage due to new release 2023-04.1

Fri, 05 May 2023 02:00:00 +0000

Outage 2023-05-05 12:00 UTC - Outage was resolved.

Outage 2023-05-05 02:00 UTC - We are deploying new Testing Farm, and it is not going super well. Expect delays and in case of unexpected errors, please restart your jobs.

[Resolved] Public Ranch - unexpected outage

Fri, 28 Apr 2023 04:00:00 +0000

Resolved 2023-04-28 10:00 UTC - Composes were updated. We are resuming the queue. If you encountered some errors, please retry your testing.

Outage 2023-04-28 08:00 UTC - There is an outage in public ranch, we need to update composes. Queue is paused. ETA 2 hours. Failed tests will need restart afterwards.

[Resolved] Public Ranch - unexpected outage

Tue, 18 Apr 2023 14:00:00 +0000

Resolved 2023-04-18 20:00 UTC - Outage was resolved, it can take some time to process all requests.

Ongoing 2023-04-18 19:00 UTC - Unfortunately, it is taken even longer as expected. ETA hopefully today.

Ongoing 2023-04-18 16:00 UTC - Unfortunately, it is taken longer as expected. ETA another 2 hours.

Outage 2023-04-18 14:00 UTC - There is an outage in public ranch, we need to redeploy. ETA 2 hours.

[Resolved] Public Ranch - expired images

Fri, 14 Apr 2023 03:00:00 +0000

Resolved 2023-04-14 09:00 UTC - Images were expired, problem should be resolved.

Outage 2023-04-14 03:00 UTC -There is an outage in public ranch, we are looking into it.

[Resolved] CentOS Stream 9 aarch64 broken in Public Ranch

Fri, 14 Apr 2023 00:00:00 +0000

Resolved 2023-04-17 20:00 UTC - Image was updated to a working one, issue is resolved \o/

Outage 2023-04-14 00:00 UTC - The CentOS Stream 9 aarch64 image is broken, all jobs will fail with GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial in pipeline.log. We are waiting for the image to refresh, until then all jobs will fail.

[Resolved] Red Hat Ranch - larger queue

Wed, 29 Mar 2023 08:00:00 +0000

Resolved 2023-03-29 10:00 UTC - Queue is back to normal, resolving.

Outage 2023-03-29 08:00 UTC - There was a large queue in the morning, we scaled the deployments and it should be processed soon.

[Resolved] Public ranch - Artemis resource ended in 'error' state

Tue, 28 Mar 2023 20:00:00 +0000

Outage 2023-03-29 20:00 UTC - Issues were resolved.

Outage 2023-03-28 20:00 UTC - We are seeing some provisioning instability, we believe we addressed the problem, please restart your jobs if needed.

[Resolved] CentOS Stream 8 broken in Public Ranch

Thu, 16 Mar 2023 08:00:00 +0000

Resolved 2023-03-20 11:00 UTC - The problem was resolved. CentOS Stream 8 changed the naming in AWS, so we were using old image. Seems affected tests are now passing \o/.

Outage 2023-03-16 08:00 UTC - We are seeing various issues with the updated CentOS Stream 8 in CI. This is due to the update to RHEL-8.7.0 bits we believe. More information will land once we analyze the problems. The requests uusually error out unexpectedly.

Some example issues:

https://artifacts.dev.testing-farm.io/d5256819-3f06-4dc4-bf01-923c135de16a/

[Resolved] Public Ranch - large queue, expect delays.

Wed, 25 Jan 2023 10:00:00 +0000

Resolved 2023-01-25 17:00:00 UTC - Whole queue was processed. No more delays are expected.

Notice 2023-01-25 15:00:00 UTC - We are about 60% done, workers scaled, we expect 1-2 hours to process all the queues.

Notice 2023-01-25 10:00:00 UTC - We are rerunning 1.2k jobs due to the outage. It might take some time to process.

[Resolved] Public Ranch outage - requests stuck in queued

Tue, 24 Jan 2023 15:00:00 +0000

Resolved 2023-01-25 10:00:00 UTC - Issue was resolved. All affected users should retry testing against Public ranch.

Outage 2023-01-24 15:00:00 UTC - Due to a deployment change all requests stuck in queued state.

[Resolved] Public Ranch Outage - artifacts storage out of inodes

Tue, 22 Nov 2022 14:00:00 +0000

Follow-up - It might be needed to restart your jobs, they might have failed without logs, as the artifacts were not correctly archived.

Resolved 20:00 UTC - The issue was identified and workarounded. We got out of inodes on the 10TB storage. Follow up in TFT-1735.

Outage 14:00 UTC - The artifacts.dev.testing-farm.io is out of inodes, we are looking into cleaning it up and understanding why it happened.

[Resolved] Public Ranch outage - Copr API v2 is deprecated

Thu, 22 Sep 2022 06:00:00 +0000

Resolved - Issue was hotfixed on all ranches via https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/223.

Update 13:00 - To mitigate consequences of the outage, whole Public Ranch was suspended.

Outage 08:00 - Copr deprecated API v2 today. We are working on migration to API v3. This affects all requests which test copr builds. All requests from Packit are affected.

[Resolved] CentOS Stream Koji Outage

Thu, 11 Aug 2022 08:00:00 +0000

Resolved The issue was identified and resolved by the CentOS Stream infrastructure team.

Outage 2022-08-11 08:00 UTC - The CentOS Stream Koji is down https://lists.centos.org/pipermail/centos-devel/2022-August/120524.html. No ETA available currently.

[Resolved] CentOS Stream 9 repositories broken in CI

Thu, 16 Jun 2022 09:00:00 +0000

Updated 2022-06-16 15:00:00 UTC - Things look back to normal. Resolving the outage.

Updated 2022-06-16 10:00:00 UTC - The underlying infra problem had been fixed alreday by CentOS infra team, but it might take some time for mirrors to pick up the fixes. More info in https://pagure.io/centos-infra/issue/812. Seems the issues is affecting only Cs9, other supported composes should not be affected. We will be monitoring the problem and update the status once things settle.

Outage 2022-06-16 9:00:00 UTC - We are seeing a lot of errors from CentOS Stream 9 related to broken hashes in repositories. The root cause is currently unknown, reported to centos-infra in https://pagure.io/centos-infra/issue/812

[Resolved] rpminspect fails with "internal libpod error"

Tue, 08 Feb 2022 12:00:00 +0000

Resolved 2022-04-09 - Issue was resolved by updating to a podman pre-release which is fixing the bug.

Update 2022-03-25 - Upstream has a fix for the problem. We are waiting for a custom module build with the fix, and we will update our workers with it.

Update 2022-02-25 - We have found a minimal reproducer - https://github.com/containers/podman/issues/13227#issuecomment-1050997458

Update 2022-02-25 - Investigation is still ongoing, we are not able to reproduce the problem manually. We will try today to update to podman 4.0.1 to see if it helps with the issue.

Update 2022-02-14 - Issue was filled to podman - https://github.com/containers/podman/issues/13227

Investigating - We’re aware of an issue affecting rpminspect and causing some of the tests blow up on errors like “Error: timed out waiting for file /var/lib/containers/storage/overlay-containers/4dbddac31d1a5529d055922e3fca8d200585b5ca562e7eb9976f32d7dd96b8f2/userdata/1e05be2210ea6433b84bba70c720b55586d4987fde7005428492e583d8ebc3a7/exit/4dbddac31d1a5529d055922e3fca8d200585b5ca562e7eb9976f32d7dd96b8f2: internal libpod error”

[Resolved] Fedora CI outage - AWS images are gone

Sat, 05 Feb 2022 20:00:00 +0000

Update - We uploaded our own images to overcome the problems with Fedora Infrastructure. The outage is resolved. Please restart jour tests jobs if you need them.

Update - Because still no ETA on fixing Fedora CI provided AWS images, we are trying to upload them to AWS ourselves.

Update - We found out all images are gone after the weekend from Fedora AWS - https://pagure.io/fedora-infrastructure/issue/10532. We are working with Fedora admins to get things in good shape, but there is currently no ETA as we might need to wait for US folks to come to work.

[Resolved] Broken rpmdeplint and rpminspect in Fedora CI

Mon, 31 Jan 2022 22:00:00 +0000

Update - The underlying problem was fixed. Rpminspect and RPMdeplint works again.

Update - We are working on fixing the problem, it was broken with our migration from CentOS-8 to CentOS-Stream-8.

Investigating - We’re aware of an issue affecting rpmdeplint and rpminspect on Public Ranch.

[Resolved] Large queue times on Public Ranch

Mon, 03 Jan 2022 17:00:00 +0000

Update - We now have better scallable workers on Public Ranch and they are dynamic and detected from the deployed infrastructure provider \o/

Update - We finally fixed all the issues with the new deployment and are scaling to cope up with the 1.6k+ jobs queued up. Things should be better in few hours!

Investigating - We’re aware of an issue affecting queue times on Public Ranch of Testing Farm. Testing can be delayed several hours. Some old jobs might have failed. We are scaling the infrastructure to better cope with the high load.

[Resolved] Issues Testing Fedora Rawhide

Sun, 19 Dec 2021 20:00:00 +0000

Update - The incident has been resolved. Root cause of the problem is Bug 2034336. We rolled back to Fedora Rawhide image from 2021-12-12 which does not contain the issue.

Investigating - We’re aware of an issue affecting all testing on Fedora Rawhide. It appeared as we upgraded to the latest image 2021-12-17.