BaseOS CI on Testing Farm Status

[Resolved] CTS outage affecting Testing Farm, expect queue

Mon, 01 Jun 2026 09:55:00 +0000

Outage | 2026-06-01 09:55:00 UTC

Outage of CTS is affecting Testing Farm Beaker testing. We disabled queue processing and hoping for a quick fix. Expect delays.

Update | 2026-06-01 10:39:00 UTC

The issue is resolved, we are confirming and resuming the queue.

Resolved | 2026-06-01 10:45:00 UTC

Resolved. CTS was brought back to life.

[Resolved] Composes not update in Testing Farm

Wed, 20 May 2026 07:00:00 +0000

Outage | 2026-05-20 07:00:00 UTC

Unfortunately ITUP is broken and our Jenkins is down hosting the compose-update process. No ETA for now :(

Update | 2026-05-21 22:46:00 UTC

Jenkins is back, we should see fresh compose update during Friday.

Update | 2026-05-24 23:45:00 UTC

Jenkins is back, but the composes had not been yet updated, we are working it out.

Update | 2026-05-25 06:38:00 UTC

Compose update is in progress, next update in 2 hours

Resolved | 2026-05-26 03:37:00 UTC

The issue was resolved. Composes are now updated.

[Resolved] Red Hat ranch requests are broken due to Artemis

Mon, 04 May 2026 12:30:00 +0000

Outage | 2026-05-04 12:30:00 UTC

Our provisioner went down. We are investigating why. RH ranch queue was suspended but a lot of jobs ended in error state.

Resolved | 2026-05-05 06:20:00 UTC

The outage is resolved. Seems everything is back to normal now.

[Resolved] BaseOS CI is not triggering

Thu, 30 Apr 2026 11:57:00 +0000

Outage | 2026-04-30 11:57:00 UTC

Due to various issue the triggering of BaseOS CI broke down. We are working on resuming the triggering and will be restarting from 30th April to deal with the outage.

Update | 2026-05-04 15:34:00 UTC

Triggering whould be working now. We are working on rerun all the missed triggers.

Resolved | 2026-05-12 12:15:00 UTC

This issue was resolved, all affected jobs restarted.

[Resolved] RHEL-9.9, RHEL-10.1, RHEL-10.2 and RHEL-10.3 compose update blocked

Thu, 19 Feb 2026 10:22:00 +0000

Outage | 2026-02-19 10:22:00 UTC

RHEL-9.9.0, RHEL-10.1, RHEL-10.2 and RHEL-10.3 are blocked due to multiple issues. See https://issues.redhat.com/browse/TFT-4367 for details. No ETA yet, but the issues are escalated as blockers.

Resolved | 2026-02-23 23:32:00 UTC

Resolving, package mode images are now available.

[Resolved] Red Hat Ranch and BaseOS CI not working due to provisioner outage

Wed, 28 Jan 2026 00:00:00 +0000

Outage | 2026-01-28 00:00:00 UTC

Provisioner does not work, blocking processing all requests on Red Hat Ranch and BaseOS CI.

Update | 2026-01-28 09:00:00 UTC

The outage was resolved, the queue is going down.

Resolved | 2026-01-28 12:00:00 UTC

The queue is now gone.

[Resolved] IBM Cloud outage

Wed, 21 Jan 2026 00:00:00 +0000

Outage | 2026-01-21 00:00:00 UTC

Due to various problems with IBM Cloud, we have had to disable it’s usage temporarily until these problems are resolved. Machines from Beaker will be used and you will notice a significant delay completing request for the architectures: s390x, ppc64le

Resolved | 2026-01-29 17:55:00 UTC

Outage was resolved. Pool enabled back.

[Resolved] BaseOS CI fails to send messages to UMB

Tue, 20 Jan 2026 12:27:00 +0000

Outage | 2026-01-20 12:27:00 UTC

BaseOS CI fails to send any UMB message. It blocks all of the testing. The issue is reported and we are monitoring the progress.

Resolved | 2026-01-20 13:56:00 UTC

The outage seems resolved. We are restarting affected builds. If we miss yours - please reach out to us.

[Resolved] UMB down

Sat, 10 Jan 2026 09:00:00 +0000

Outage | 2026-01-10 09:00:00 UTC

Issue with signing server blocked umb, now cert is fixed, but UMB is overloaded

Resolved | 2026-01-13 23:00:00 UTC

UMB is running, queue has been processed and BaseOS-CI works normaly again

[Resolved] Provisioning errors due to missing s390x images

Fri, 12 Dec 2025 10:26:00 +0000

Outage | 2025-12-12 10:26:00 UTC

Our compose-update let through some composes with missing s390 images in IBM cloud, breaking provisioning. We are looking into the situation, for more information see: https://redhat-internal.slack.com/archives/C03BRN71JAF/p1765792914415559

Resolved | 2025-12-15 12:25:00 UTC

Errors resolved and images synced.

[Resolved] Beaker outage affecting provisioning

Tue, 09 Dec 2025 07:59:00 +0000

Outage | 2025-12-09 07:59:00 UTC

We are seeing elevated error rate when provisioning Beaker, investigation is in progress.

Update | 2025-12-09 08:40:00 UTC

Seems things look more stable now, reported to #help-beaker channel for investigation of the 500s we and users directly interacting with Beaker API were seeing: https://redhat-internal.slack.com/archives/C04KRPW1HJ4/p1765266395518069

Update | 2025-12-09 18:26:00 UTC

Resolved, it was caused by Beaker going out of disk space :(

Resolved | 2025-12-09 20:06:00 UTC

Resolving …

[Resolved] Testing Farm updating AWS DNS setup to support DNSSEC, failover to Openstack and Beaker

Thu, 06 Nov 2025 21:00:00 +0000

Outage | 2025-11-06 21:00:00 UTC

For more information see: https://groups.google.com/a/redhat.com/g/rhel-devel/c/-46R4axLj_s

Resolved | 2025-11-06 23:45:00 UTC

The setup is complete, unfortunately the changes did not have effect we hoped for. See https://redhat-internal.slack.com/archives/C03BRN71JAF/p1762277234611719 Slack for more information. Resolving the outage.

[Resolved] Beaker pools not working

Fri, 31 Oct 2025 09:05:00 +0000

Outage | 2025-10-31 09:05:00 UTC

Due to a breaking change in Artemis v0.0.81 the Beaker pools became broken and erroring out all the time.

Update | 2025-11-03 09:07:00 UTC

The problem was identified early Monday, and the fix is getting deployed to mitigate this problem. Next to it we are extending our monitoring to cover the problem.

Update | 2025-11-03 10:44:00 UTC

The issue is resolved, we are monitoring the problem.

Update | 2025-11-04 03:15:00 UTC

Issue should be resolved

Resolved | 2025-11-06 20:52:00 UTC

Resolving, things are stable now.

[Resolved] AWS world-wide outage affecting Testing Farm

Mon, 20 Oct 2025 09:29:00 +0000

Outage | 2025-10-20 09:29:00 UTC

The world wide AWS outage is affecting Testing Farm. We are looking how to mitigate the issue, but be prepared for delays in testing. Public Ranch is unaffected as it is hosted in us-east-2.

Update | 2025-10-20 18:00:00 UTC

Issues are persisting, things looked better, now it is worse and we had to disable AWS alltogether. Expect delays.

Resolved | 2025-10-21 01:04:00 UTC

Outage is resolved. AWS should be back to normal operations. We have enabled back the AWS pools in general routing.

[Resolved] Brew outage

Fri, 17 Oct 2025 09:00:00 +0000

Outage | 2025-10-17 09:00:00 UTC

Maintenance affecting brew, dist-git, MBS

Update | 2025-10-20 07:19:00 UTC

Brew is working again, resolving.

Resolved | 2025-10-20 09:11:00 UTC

Resolving

[Resolved] Fedora and CentOS stream compose broken due to download.bos down

Thu, 02 Oct 2025 10:40:00 +0000

Outage | 2025-10-02 10:40:00 UTC

We are aware of an issue affecting Fedora and CentOS Stream testing in Red Hat ranch. This is due to download.bos outage, we are working on a workaround.

Resolved | 2025-10-08 23:22:00 UTC

This is resolved, problem was addressed

[Resolved] BaseOS-CI issue with UMB

Sun, 28 Sep 2025 22:00:00 +0000

Outage | 2025-09-28 22:00:00 UTC

dispatchers are blockd on writing to UMB

Resolved | 2025-09-29 22:29:00 UTC

The issues had been resolved. Jobs are resuming. No outage except the delay is expected. Let us know if you would need some assitance.

[Resolved] Wow fails on connection to TCMS in BaseOS CI

Fri, 26 Sep 2025 10:54:00 +0000

Outage | 2025-09-26 10:54:00 UTC

We are aware of an issue in TCMS affecting some test jobs using workflow-tomorrow (wow). The error usually looks like [2025-09-15 13:54:42] [E] [test-scheduler-artemis] Exception raised: Failure during 'wow' execution: and the wow debug log shows http.client.IncompleteRead: IncompleteRead(48586 bytes read) at the end. This is caused by https://gitlab.cee.redhat.com/sustaining-engineering/tcms/-/issues/2 and the TCMS folks are looking into it.

Resolved | 2025-10-08 23:22:00 UTC

This was resolved, tcms is stable now.

[Resolved] BaseOS-CI jenkins controller migration

Mon, 22 Sep 2025 22:00:00 +0000

Outage | 2025-09-22 22:00:00 UTC

suspended queue, after some time remaining jobs will be canceled and restarted on new location

Resolved | 2025-09-23 01:30:00 UTC

BaseOS-CI jenkins controller migrated and fully operational.

[Resolved] BaseOS CI is unstable

Mon, 08 Sep 2025 02:00:00 +0000

Outage | 2025-09-08 02:00:00 UTC

BaseOS CI is down from early morning. We are looking into the issue.

Update | 2025-09-09 09:00:00 UTC

It was discovered that a Beaker outage is affecting various infrastructure components. Some issues have been resolved, others are still being worked on.

Update | 2025-09-11 08:13:00 UTC

We are experiencing frequent short downtimes of Jenkins. We are trying to resolve the issue.

Update | 2025-09-19 09:40:00 UTC

The issues is still present, we are still working to resolve the issue

Resolved | 2025-09-23 01:44:00 UTC

migration complete, jenkins is up and crunching tasks

[Resolved] Beaker provisioning issues

Mon, 01 Sep 2025 11:50:00 +0000

Outage | 2025-09-01 11:50:00 UTC

There is a wide problem for Beaker causing provisioning issues. Affecting at least s390x. See https://redhat.service-now.com/surl.do?n=INC3762615 for details.

Resolved | 2025-09-03 17:32:00 UTC

Outage was resolved in the evening of 3rd September

[Resolved] BaseOS CI failing on wow

Thu, 21 Aug 2025 17:00:00 +0000

Outage | 2025-08-21 17:00:00 UTC

Due to the TCMS migration, the old connection URL does not work anymore and BaseOS CI is not able to connect to TCMS via wow. We are working out the issue.

Resolved | 2025-09-01 11:50:00 UTC

BaseOS CI worker image was updated to latest qa-tools, effectively resolving the issue.

[Resolved] Red Hat Gitlab Access Limited

Wed, 13 Aug 2025 09:00:00 +0000

Outage | 2025-08-13 09:00:00 UTC

The Gitlab team is workning to resolve this. You may see HTTP 429 responses in your logs and timeouts. If urgent, please restart the test job.

Resolved | 2025-08-14 07:03:00 UTC

Seems the outage is resolved. Gitlab.cee and pipelines work now well.

[Resolved] baseos-ci queue

Fri, 08 Aug 2025 08:17:00 +0000

Outage | 2025-08-08 08:17:00 UTC

BaseOS-CI have issues with auto-scaling, leading to long queue. It will take at least few hours to process the queue

Resolved | 2025-08-18 08:38:00 UTC

Resolving this old outage. The queue was resolved during the day of outage.

[Resolved] BaseOS-CI stuck pipelines

Thu, 24 Jul 2025 14:45:00 +0000

Outage | 2025-07-24 14:45:00 UTC

last 2 hours we have huge load leading to either provisioning issues or jenkins instability

Update | 2025-07-28 09:09:00 UTC

during weekend, jenkins stopped processing jobs completely. Now it’s restarted and workers are building

Update | 2025-07-28 09:50:00 UTC

jenkins is fully operationl, workers are crunching new jobs. Queue from weekend was corrupted, I will try to reschedule those jobs

Update | 2025-08-02 09:35:00 UTC

most of affected jobs were restarted and processed. Some missed dispatcher to be retriggered

Resolved | 2025-08-18 09:11:00 UTC

Resolved for a long time.

[Resolved] Compose update delayed due to gitlab workers remaining in pending state.

Wed, 23 Jul 2025 09:15:00 +0000

Outage | 2025-07-23 09:15:00 UTC

Intermitent issue. May have minimal impact, but will affect users requiring today’s composes.

Resolved | 2025-07-23 17:26:00 UTC

gitlab workers are running jobs again now with no action from the Testing Farm team.

[Resolved] BaseOS CI and Testing Farm testing blocked on artifacts.osci.redhat.com outage

Wed, 16 Jul 2025 22:00:00 +0000

Outage | 2025-07-16 22:00:00 UTC

The artifacts.osci.redhat.com was migrated to ITUP infrastructure, and the NFS shares did not make it. Basically archiving is impossible, and that is making all the jobs blocked. No ETA currently, folks will be looking into it in the morning.

Update | 2025-07-17 09:25:00 UTC

We are working with ITUP to revive the artifacts storage. No ETA available yet.

Update | 2025-07-17 14:35:00 UTC

We are working with OSCI and ITUP on creating a new NFS volume and migrating a part of artifacts data. No ETA available yet.

Update | 2025-07-17 19:00:00 UTC

The investigation and work with OSCI and ITUP continues. Unfortunately, no ETA on resolving the issue yet :(

Update | 2025-07-18 08:00:00 UTC

The artifact storage has been migrated once again, and the NFS response is back to where it needs to be. Both Red Hat Ranch and Base OS CI were unblocked and started crunching their queue. We are monitoring the situation before the outage gets resolved properly.

Resolved | 2025-07-18 10:30:00 UTC

Resolved. The storage is stable, and the queue is getting smaller.

[Resolved] OpenStack outage

Mon, 14 Jul 2025 08:00:00 +0000

Outage | 2025-07-14 08:00:00 UTC

OpenStack is not operational and affects testing on machines managed by OpenStack.

Resolved | 2025-07-15 21:00:00 UTC

Image listing problem found and resolved.

[Resolved] BaseOS CI artifacts upload not working

Sat, 12 Jul 2025 08:00:00 +0000

Outage | 2025-07-14 07:54:00 UTC

Due to migration of artifacts.osci.redhat.com to ITUP we are not able to upload logs from BaseOS CI to the artifacts storage. This makes some logs unvailable, but otherwise BaseOS CI should be working. We are working out the issue with OSCI, but currently there is no ETA.

Resolved | 2025-07-14 10:10:00 UTC

The outage was resolved via https://redhat.service-now.com/help?id=rh_ticket&table=incident&sys_id=c146c1a8fb366e18ece2f7744eefdc32

[Resolved] BaseOS CI auto-scaling broken, testing stuck

Mon, 30 Jun 2025 09:08:00 +0000

Outage | 2025-06-30 09:08:00 UTC

BaseOS CI testing is currently blocked, because autoscaling does not work, we are looking into the issue.

Resolved | 2025-06-30 10:13:00 UTC

Resolved, the autoscaling still nto working, but we were able to create a worker manually.

[Resolved] BaseOS CI stability issues

Tue, 17 Jun 2025 08:24:00 +0000

Outage | 2025-06-17 08:24:00 UTC

Note that we are fighting with BaseOS CI stability due to the migration to ITUP. If you encounted any issues please contact us on #testing-farm Slack thread.

Update | 2025-06-20 12:52:00 UTC

Seems we are more stable now, keeping the outage open for a few more days.

Resolved | 2025-06-30 09:08:00 UTC

This issue was resolved, BaseOS CI should be more stable now.

[Resolved] Red Hat Ranch and BaseOS CI not working due to provisioner outage

Fri, 13 Jun 2025 22:00:00 +0000

Outage | 2025-06-13 22:00:00 UTC

Provisioner does nto work, blocking processing all requests on Red Hat Ranch and BaseOS CI

Update | 2025-06-15 11:29:00 UTC

The root cause of the issue was identified and hotfixed. We have a fix so we mitigate this issue to happen again. If you encounter any unexpected provisioning or timeout errors in your test requests, please restart them via your CI system or manually.

Resolved | 2025-06-17 08:23:00 UTC

The outage was resolved on 15th, closing outage.

[Resolved] IT resilience testing multiple SP services down

Fri, 13 Jun 2025 13:00:00 +0000

Outage | 2025-06-13 13:00:00 UTC

tps-server

Resolved | 2025-06-13 21:00:00 UTC

Testing complete

[Resolved] BaseOS-CI restart

Tue, 10 Jun 2025 20:17:00 +0000

Outage | 2025-06-10 20:17:00 UTC

To update jenkins controller configuration, I need to restart jenkins. Queue suspended.

Resolved | 2025-06-10 20:46:00 UTC

jenkins restarted and working

[Resolved] Disabled direct beaker pipelines in BaseOS-CI

Tue, 10 Jun 2025 12:20:00 +0000

Outage | 2025-06-10 12:20:00 UTC

there is high load in beaker, instability in websocket between jenkins agents and controller and beaker monthly updates. Disabling jenkins jobs for beaker.

Update | 2025-06-10 13:08:00 UTC

canceling all jobs owned by tft-automation-prod

Update | 2025-06-10 15:30:00 UTC

finaly canceled beaker jobs, previous attempt was blocked by beaker maintenance, restarting jenkins with some network related changes

Resolved | 2025-06-13 08:15:00 UTC

this issue seems to be fixed, pipelines enabled again

[Resolved] ppc64le testing blocked

Mon, 09 Jun 2025 07:57:00 +0000

Outage | 2025-06-09 07:57:00 UTC

becaues of long queue time in beaker, ppc64le is blocked

Update | 2025-06-09 09:43:00 UTC

many ppc64le requests running for long time canceled

Resolved | 2025-06-13 08:14:00 UTC

queue is almost empty, ppc64le testing shoud work well. In case of aborted job because of timeout, restart your tests

[Resolved] RHEL modules testing broken

Thu, 05 Jun 2025 21:00:00 +0000

Outage | 2025-06-05 21:00:00 UTC

Changed keytab cause lost of permissions for ODCS

Resolved | 2025-06-06 13:03:00 UTC

ODCS access has been granted and it works from BaseOS-CI again. All affected jobs restarted.

[Resolved] workflow-tomorrow tests will not run

Wed, 04 Jun 2025 08:00:00 +0000

Outage | 2025-06-04 08:00:00 UTC

Due to decommision of REDHAT.COM kerberos realm, workflow-tomorrow tests do not run.

Update | 2025-06-05 17:49:00 UTC

new keytab is deployed and tested, restarting affected jobs. There is ~ 300 jobs, so it will take many hours to process

Update | 2025-06-05 20:11:00 UTC

wow works again, most of affected jobs restarted, pending issue with ODCS access

Resolved | 2025-06-06 13:00:00 UTC

new keytab is working as expected

[Resolved] Beaker outage

Wed, 04 Jun 2025 07:33:00 +0000

Outage | 2025-06-04 07:33:00 UTC

Beaker machines are unavailable due to a connectivity problem. Investigation is ongoing.

Update | 2025-06-04 08:50:00 UTC

Authentication update needed.

Update | 2025-06-04 08:56:00 UTC

Authentication updated… testing.

Update | 2025-06-04 11:35:00 UTC

Confirmed working.

Resolved | 2025-06-04 11:25:00 UTC

Change deployed to production.

[Resolved] RHEL9 and RHEL10 compose update blocked on IBM cloud outage

Mon, 02 Jun 2025 13:00:00 +0000

Outage | 2025-06-02 13:00:00 UTC

IBM Cloud outage is blocking updating composes for RHEL9 and RHEL10. We are waiting for https://groups.google.com/a/redhat.com/g/outage-list/c/u93AOmuaKqM to be resolved to resume the process.

Update | 2025-06-04 10:20:00 UTC

The issue was resolved yesterday, but it is back again today :(

Resolved | 2025-06-11 08:20:00 UTC

Seems the issue is resolved.

[Resolved] Planned BaseOS-CI outage - migration

Thu, 29 May 2025 19:31:00 +0000

Outage | 2025-05-29 19:31:00 UTC

suspended queue, migration will start when there will be less running jobs

Update | 2025-05-30 01:07:00 UTC

migrated instance up and running, it will take some time to consume queue

Resolved | 2025-05-30 01:44:00 UTC

everything work as expected, queue almost consunemd

[Resolved] Artemis issue with policies

Thu, 22 May 2025 09:07:00 +0000

Outage | 2025-05-22 09:07:00 UTC

Investigating root cause and reverting recent changes. Requests are stuck in routing and will be processed after revert

Update | 2025-05-22 11:52:00 UTC

Issue fixed, queue of guests in routing is going down

Resolved | 2025-05-22 12:57:00 UTC

All requests stuck in routing are now processed. In case of timeout, please restart your test.

[Resolved] ansible-core breaks ansible-lint

Tue, 20 May 2025 07:42:00 +0000

Outage | 2025-05-20 07:42:00 UTC

Tooling affected by https://github.com/ansible/ansible-lint/issues/4611

Update | 2025-08-07 12:00:00 UTC

This has been mostly mitigated by forcing the ansible version to something before 2.19.

Update | 2025-08-18 09:12:00 UTC

Seems resolved for some time now.

Resolved | 2025-08-19 17:20:00 UTC

Resolving

[Resolved] Instances gone during test execution

Tue, 20 May 2025 06:00:00 +0000

Outage | 2025-05-20 06:00:00 UTC

The spot instances are getting lately terminated more then expected. We are moving to non-spot instances by default, see https://issues.redhat.com/browse/TFT-3371 for details.

Resolved | 2025-05-22 18:01:00 UTC

The issue should be resolved, we are now using non-spot instances by default.

[Resolved] BaseOS-CI outage - jenkins migration

Sat, 17 May 2025 11:00:00 +0000

Outage | 2025-05-17 11:00:00 UTC

Stopping jenkins and starting vm migration

Resolved | 2025-05-17 15:23:00 UTC

Normal operation resumed

[Resolved] Beaker outage

Tue, 13 May 2025 14:18:00 +0000

Outage | 2025-05-13 14:18:00 UTC

Beaker outage affects all SUT/DUTs from Beaker.

Resolved | 2025-05-14 07:02:00 UTC

Beaker reboots complete.

[Resolved] BaseOS-CI outage - migration

Sat, 10 May 2025 18:35:00 +0000

Outage | 2025-05-10 18:35:00 UTC

Another try to migrate vm. I expect several hour outage.

Resolved | 2025-05-11 09:00:00 UTC

image transfer completed, starting jenkins

[Resolved] BaseOS-CI outage - migration 2nd try

Fri, 09 May 2025 20:43:00 +0000

Outage | 2025-05-09 20:43:00 UTC

shuting down jenkins controller for vm migration

Resolved | 2025-05-10 09:25:00 UTC

jenkins controller is working again, closing outage

[Resolved] Planned BaseoOS-CI outage (part of migration)

Thu, 08 May 2025 19:00:00 +0000

Outage | 2025-05-08 12:00:00 UTC

Ongoing preparatin to minimize downtime. In case this would cause you issue, let me know (optak)

Update | 2025-05-08 19:00:00 UTC

queue suspended

Update | 2025-05-08 20:21:00 UTC

migration begins

Resolved | 2025-05-08 23:01:00 UTC

outage is over, all affected jobs restarted

[Resolved] BRQ beaker lab aborting jobs

Mon, 05 May 2025 11:00:00 +0000

Outage | 2025-05-05 11:00:00 UTC

jobs ara aborting because of disk space issue

Resolved | 2025-05-05 19:33:00 UTC

resolved by beaker team

[Resolved] BaseOS CI break with disk space hardware requirements

Thu, 17 Apr 2025 08:00:00 +0000

Outage | 2025-04-17 08:00:00 UTC

We are aware of an issue with BaseOS-CI causing the jobs with hardware requirements for disk space to fail. We are working to resolve the issue.

Resolved | 2025-04-18 08:00:00 UTC

All affected jobs were restarted. Closing outage.

[Resolved] Artifacts storage outage for BaseOS CI and Testing Farm

Fri, 11 Apr 2025 13:00:00 +0000

Outage | 2025-04-11 13:00:00 UTC

The artifacts storage was restarted and lost all mount points including Testing Farm and BaseOS CI artifacts storage.

Resolved | 2025-04-11 15:10:00 UTC

The used DNS servers were removed by IT last year. The servers were corrected, mounts points remounted and the outage should be cleared. We are planning to move away from this artifacts storage for better experience.

[Resolved] Large queue on Red Hat ranch

Mon, 07 Apr 2025 16:02:00 +0000

Outage | 2025-04-07 16:02:00 UTC

Expect larger queue, we are working on scaling up Testing Farm to deal with the queue.

Resolved | 2025-04-07 20:40:00 UTC

The queue is gone, pipeline should be back to normal.

[Resolved] RHEL-10.0 is expired on Red Hat Ranch

Mon, 07 Apr 2025 07:58:00 +0000

Outage | 2025-04-07 07:58:00 UTC

We are working to resolve the issue.

Update | 2025-04-07 09:42:00 UTC

Images are built, we are working on releasing them. ETA 1h.

Update | 2025-04-07 11:11:00 UTC

Updated compose is deployed to production.

[Resolved] Planned brew outage

Fri, 28 Mar 2025 10:00:00 +0000

Outage 2025-03-28 10:00 UTC - Start of planned Brew outage window. BaseOS-CI is suspended, Red Hat Ranch is affected for manually triggered brew task testing

Update 2025-03-28 14:15 UTC - Some complication, unplanned outage right after end of planed one

Resolved 2025-03-29 12:00 UTC - Resuming queue (meanwhile jenkins controller has been upgraded)

[Resolved] Fedora broken on aarch64, s390x and ppc64le in Beaker

Mon, 10 Mar 2025 06:00:00 +0000

Outage | 2025-03-10 06:00:00 UTC

Fedora testing on s390x, ppc64le and aarch64 is broken. Please share your concern in https://redhat.service-now.com/surl.do?n=INC3438634.

Update | 2025-05-27 08:20:00 UTC

The issue is still present. We have users who are failing to conduct their testing in Fedora-Rawhide and Fedora-42.

Update | 2025-11-03 10:43:00 UTC

This issue is still present. Please update the service now help ticket.

Resolved | 2025-11-27 15:48:00 UTC

The outage is resolved finally. The distros should be installable without issues in Beaker.

[Resolved] Beaker Outage - several issues

Wed, 12 Feb 2025 08:00:00 +0000

Outage 2025-02-12 08:00 UTC - New release of restraint broke reservation

Update 2025-02-17 20:00 UTC - Issue with restraint solved, but beaker is not in a good shape: long queues, many machines marked as broken

Update 2025-02-18 16:00 UTC - Beaker is getting well, but we have some composes outdated, new ones bloced by issue with installing qa-tools

Resolved 2025-02-19 09:00 UTC - All expired composes updated, everything works as expected, beaker queues < 4h

[Resolved] Brew outage https://brewweb.engineering.redhat.com/brew/

Tue, 10 Dec 2024 09:00:00 +0000

Outage 2024-12-19 11:30:00 UTC - Brew service is down. All testing relying on Brew are affected. See https://brewweb.engineering.redhat.com/brew/

[Resolved] download.devel SSL certificate expired

Sun, 24 Nov 2024 21:00:00 +0000

Resolved 2024-11-25 10:30 UTC - Outage was resolved by IT. Certificate was refreshed.

Outage 2024-11-24 23:00 UTC - See https://redhat.service-now.com/surl.do?n=INC3256074 for details.

Outage 2024-11-24 21:00 UTC - The download.devel SSL certificate has expired, this is affecting BaseOS CI and Testing Farm Red Hat ranch.

[Resolved] Red Hat Ranch / BaseOS CI Outage - provisionig issues during the weekend

Sat, 09 Nov 2024 00:00:00 +0000

Resolved 2024-11-11 10:00 UTC - Queues are consumed, outage is over.

Outage 2024-11-11 09:00 UTC - We are working on resolving the issue in about an hour.

Outage 2024-11-10 00:00 UTC - Requests are blocked and the queue is rising. All requests will timeout out and restarting will be needed after the weekend.

Outage 2024-11-09 00:00 UTC - Due a bug causing the Artemis worker tasks to wait forever to extend reservations in PEK labs, our provisioner went down.

[Resolved] Gitlab CEE outage affecting jobs run on BaseOS CI and Red Hat ranch.

Mon, 04 Nov 2024 18:00:00 +0000

Resolved 2024-11-05 03:00 UTC - The outage is over. See outage list for more details.

Outage 2024-11-04 18:00 UTC - GitLab CEE outage affecting a lot of jobs.

[Resolved] BaseOS CI and Red Hat Ranch affected by provisioning outages. BaseOS CI down.

Fri, 01 Nov 2024 00:00:00 +0000

Resolved 2024-11-01 09:00 UTC - Outage was resolved. Please restart jobs if ended in error state.

Outage 2024-11-01 00:00 UTC - BaseOS Jenkins is down, and Red Hat ranch provisioning ending in error state.

[Resolved] BaseOS CI and Red Hat Ranch affected by OpenShift outage

Thu, 31 Oct 2024 12:00:00 +0000

Partial outage 2024-10-31 12:00 UTC - BaseOS CI and Red Hat Ranch affected by maintenance of RDU2 OpenShift.

[Resolved] BaseOS CI suspended for changes in UMB certificate.

Tue, 15 Oct 2024 17:00:00 +0000

**Resolved 2024-10-15 23:55 UTC - queue resumed, still running jobs canceled & restarted

Outage 2024-10-15 17:00 UTC - queue suspended, except for dispatchers

Planned maintenance 2024-10-15 14:00 UTC - Planned maintenance of jenkins controllers. Impact: suspended queue since 17:00 UTC, expected end: 20:00 UTC

[Resolved] Red Hat Ranch / BaseOS CI Outage - provisionig issues

Tue, 03 Sep 2024 20:00:00 +0000

Outage 2024-09-04 02:00 UTC - We were not able to resolve the issues during night, your requests might error out or get canceled.

Outage 2024-09-03 20:00 UTC - We are experiencing provisioning issues on Red Hat ranch.

[Resolved] Red Hat Ranch and BaseOS-CI Outage - dns misconfiguration, some crucial sites not accessible

Tue, 13 Aug 2024 15:40:00 +0000

Resolved 2024-08-14 00:50 UTC - both services fully operational.

Update 2024-08-13 23:42 UTC - dns issues looks resolved, baseosci resumed (long queque), testing farm under testing

Outage 2024-08-13 15:50 UTC - because of issue with dns servers, some internal sites are not accessible, BaseOS-CI is suspended to lower impact

[Resolved] Beaker PPC64 Outage - large queue because of limited hardware

Fri, 26 Jul 2024 00:00:00 +0000

Resolved 2024-08-02 00:00 UTC - ppc64le queue times stayed below 4 hours for 3 days.

Outage 2024-07-26 00:00 UTC - due to a deteriorating PPC64 pools in Beaker, testing on this ARCH is not advised. Expect over 24 hours delay.

[Resolved] Public Ranch - Large queue

Mon, 17 Jun 2024 13:00:00 +0000

Warning 2024-06-17 13:00 UTC - Public ranch has 1k+ queued jobs, seems Fedora CI load, be patient, it is getting progressed, but expect delays.

[Resolved] Red Hat Ranch and BaseOS-CI Outage - broken krb infra

Tue, 28 May 2024 18:00:00 +0000

Resoled 2024-05-29 10:00 UTC - Change related to krb infra reverted, everything works now.

Update 2024-05-29 9:00 UTC - BaseOS-CI partially unblocked by deploying workaround (missing fix in artemis)

Update 2024-05-29 10:00 UTC - something has changed in kerberos infra leading to broken keytab. All requests to beaker are affected

[Resolved] BaseOS CI suspended for urgent jenkins controller upgrade.

Fri, 02 Feb 2024 08:00:00 +0000

Resolved 09:00 UTC - Jenkins controller is running, all affected jobs were restarted.

Outage 08:00 UTC - All jobs (except for dispatchers) suspended as a preparation for jenkins controller upgrade. Expect delays for test result.

[Resolved] BaseOS CI suspended for urgent jenkins controller upgrade.

Wed, 03 Jan 2024 10:47:00 +0000

Resolved 12:30 UTC - Jenkins controller is running, all affected jobs were restarted.

Outage 10:47 UTC - All jobs (except for dispatchers) suspended as a preparation for jenkins controller upgrade. Expect delays for test result.

[Resolved] BaseOS CI limited due to outage due to BRQ network maintanance.

Sun, 12 Nov 2023 00:00:00 +0000

Resolved 12:00 UTC - Registry is back, outage is resolved. The tests should automatically resume, no manual intervention should be needed.

Outage 00:00 UTC - Our BRQ image registry went down after BRQ network maintanance. This is blocking BaseOS CI autoscaling, so the queue can get large. We are working to resolve the issue.

[Resolved] BaseOS CI and Testing Farm Red Hat Ranch outage due to Beaker unavailability

Tue, 07 Nov 2023 15:00:00 +0000

Resolved 19:00 UTC - Beaker is back, we resumed BaseOS CI and Testing Farm, jobs are processing.

Outage 15:00 UTC - Beaker is down including required repositories which break all executions on RH ranch for RHEL. We disabled the ranch completely to overcome too many failures. Will be posting updates here.

[Resolved] Testing Farm Red Hat ranch and BaseOS CI outage for RHEL9 and RHEL-8.1.0

Mon, 25 Sep 2023 00:00:00 +0000

Resolved 16:30 UTC - All affected composes updated, the problems should be gone. We will try to auto-restart fail jobs, please feel free to restart yourself also if in hurry.

Update 12:30 UTC - All problems affecting building had been fixed. We are getting fresh composes through. ETA 2-3 hours.

Outage 0:00 UTC - Composes have expired due to a sad misunderstanding from Friday :( We are trying to resolve the issue, but it will take few hours.

[Resolved] BaseOS CI outage due to network issues

Thu, 21 Sep 2023 00:00:00 +0000

Resolved 7:00 UTC - Problem was resolved by IT. BaseOS CI is back in action.

Outage 0:00 UTC - After restart DHCP does not resolve address of the machine. Working with IT to investigate the problem.

[Resolved] Beaker testing blocked on Red Hat Ranch and BaseOS CI

Mon, 21 Aug 2023 08:00:00 +0000

Resolved 12:00 UTC - The issue was resolved by Beaker admins. Outage is resolved.

Update 9:00 UTC - Seems the issue is only with /distribution/reservesys, the task was disabled in Beaker :) We are waiting for admins to resolve.

Outage 8:00 UTC - All testing against Beaker is blocked due to missing /distribution tests from Beaker. Investigation is ongoing.

[Resolved] Red Hat Ranch & BaseOS CI - outage due to elevated cloud costs

Tue, 02 May 2023 08:00:00 +0000

Resolved 2023-05-02 12:00 UTC - Problem was resolved. Queue was resolved, requests should start resuming, no issues are expected.

Outage 2023-05-02 08:00 UTC - We stopped BaseOS CI and Testing Farm Red Hat ranch, because we see Artemis provisioning c5.24xlarge machines, we are investigation why. This is causing elevated costs.

[Resolved] Fedora-Rawhide broken in Red Hat Ranch

Fri, 10 Mar 2023 08:00:00 +0000

Resolved 2023-03-16 12:00 UTC - We have found the issue, Rawhide is broken on Xen instances. Filled to Fedora infrastructure: https://pagure.io/fedora-infrastructure/issue/11185. We workarounded by using nitro instances for the testing.

Outage 2023-03-10 08:00 UTC - Fedora Rawhide is broken in Red Hat ranch and BaseOS CI. The instance does not boot. Requests will error out. Currently the root cause is unknown and not ETA for resolving.

[Resolved] Red Hat internal infrastructure outage

Fri, 10 Feb 2023 09:00:00 +0000

Outage 18:00 UTC 11th Feb - All processing resumed, long queue for next few hours.

Outage 16:00 UTC - Requests to Red Hat ranch might fail unexpectedly due to the internal outage. The RHEL images depend on some of the infrastructure being down - namely repositories to beakerlib and such. The outage should last until Fri, 10 Feb 2023 22:00:00 UTC.

Outage 09:00 UTC - We paused processing of all jobs via BaseOS CI, to mitigate outage effects.

[Resolved] BaseOS CI - large queue, expect delays.

Mon, 30 Jan 2023 16:00:00 +0000

📓 Notice 2023-02-13 18:00:00 UTC - We believe things are now a lot better, we are able to keep with the queue. Resolving.

📓 Notice 2023-01-30 16:00:00 UTC - We are experiencing a larger queue on BaseOS CI. This can affect baseos-ci.* and leapp.* tests. Delays are expected. We are having trouble scaling the deployment due to some bugs in the deployment :(

[Resolved] BaseOS CI Outage

Tue, 06 Dec 2022 08:00:00 +0000

Resolved 17:00 UTC - The issue was resolved. We are restarting all affected jobs. It might take some time for all jobs to process.

Outage 15:00 UTC - We still do not have an ETA here, if the issue would persist, we will workaround it. Expecting to resume jobs this night.

Outage 12:00 UTC - We paused processing of all jobs, to mitigate outage effects and discussing how to mitigate this problem in the future.

Outage 08:00 UTC - BaseOS CI is currently failing some of it’s test jobs due to an internal outage. More info in TFT-1770.

[Resolved] BaseOS CI Outage - UMB message sending

Mon, 31 Oct 2022 08:00:00 +0000

Resolved 12:00 UTC - The issue was resolved, the UMB cert was rotated. We are restarting all affected jobs. Processing the queue can take some time. Contact us in case of trouble.

In Progress 10:30 UTC - The issue was identified - an expired UMB certificate. We are working with IT to be able to rotate it. Current ETA is at least 2 more hours.

Outage 8:00 UTC - BaseOS CI is not processing any jobs currently, it is failing to send out any UMB message. We are investigating the problem.

[Resolved] BaseOS CI Outage

Fri, 12 Aug 2022 18:00:00 +0000

Resolved - The issue was resolved by dropping the requirement for the bits stored in BRQ.

Outage - BaseOS CI is not processing any jobs currently, as it is broken due to a dependency from BRQ, where the LAB is being moved. We are working hard to get it back to processing jobs.

[Resolved] Covscan testing blocked on BaseOS CI

Fri, 25 Mar 2022 12:00:00 +0000

Resolved - covscan testing on BaseOS CI was resumed. Outage is over.

Outage - covscan is boggled with too many scans from Errata due to a known issue. We will be blocking testing until covscan is freed up and can peform CI testing. Currently there is no ETA.

[Resolved] Expired RHEL-8.6.0 and RHEL-9.0.0 images in Red Hat Ranch and BaseOS CI

Wed, 29 Dec 2021 11:00:00 +0000

Update - Images were updated, failed tests restarted. In case you encounter issues, please restart your failed jobs.

Update - Red Hat tracking issue

Investigating - We’re aware of expired images for RHEL-8.6.0 and RHEL-9.0.0. They are causing failures for installability test and all functional tests. We are working to resolve the problem ASAP and will try to restart all affected jobs once the issue is resolved.