12:30 <walters> Just thinking about how the periodic race conditions between koji tagger and yum repo sync would be fixed if we versioned rpm-md repositories as OCI artifacts instead (hit this in https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/ostree/detail/PR-2926/2/pipeline )

12:55 <vrothberg> dustymabe: the change to the ready.service of podman-machine considerably improved the flake rate (see github.com/containers/podman/issues/17403). But ... I am still seeing the flake. What I observe is that always the first SSH attempt into the machine fails. That happens right after starting the machine (and receiving the signal from the ready.service).

12:55 <vrothberg> Adding a sleep etc. resolves the issue. It seems there is still a race. Do you have any other tricks in your pockets? I am tempted to just do an exponential backoff to "wait" for SSH in the machine to be ready. But that feels like patching symptoms rather than fixing the underlying issue.

13:20 <dustymabe> vrothberg: I don't have any other ideas without spending more time digging into the problem in depth. It all depends on where the error is happening. Is it actually reaching the ssd process or is port 22 somehow not yet bound? If it's the former we'd need to look at the source code.

13:20 <dustymabe> sometimes a sleep/retry is ugly, but it's practical. and at least the delay here should be really small.

13:22 <vrothberg> dustymabe: Thanks for checking! I agree that a client-side check is probably the best way forward at the moment.

13:27 <dustymabe> vrothberg: sorry I couldn't be of more help :(

13:27 <dustymabe> marmijo[m]: mind a review on: https://pagure.io/fedora-infra/ansible/pull-request/1505

13:28 <vrothberg> dustymabe: you were of great help! You reassured me that a client-side retry is a valid approach, so I am happy :)

13:33 <dustymabe> walters: we put a mitigation for this in the pipeline (adamP used it as a learning opportunity): https://github.com/coreos/fedora-coreos-pipeline/pull/863

13:34 <dustymabe> as a followup we were going to try to make it more generic and maybe hoist it up into coreos-ci-lib so the upstream CI projects would have the workaround

14:46 <dustymabe> marmijo[m]: I think the vexxhost issues settled since yesterday

15:26 <marmijo[m]> dustymabe: That's good. I see we've had some successful kola openstack jobs.

15:37 <marmijo[m]> bgilbert: It looks like the new release of ignition in rawhide is attempting to install `hv_utils` on ppc64le and s390x and causing the builds for those arches to fail.

16:30 <dustymabe> aaradhak anthr76 apiaseck davdunc dustymabe guidon gursewak jaimelm jbrooks jcajka jdoss jlebon jmarrero lorbus miabbott nasirhm quentin9696[m] ravanelli saqali walters

16:30 <dustymabe> FCOS community meeting in #fedora-meeting-1

16:30 <dustymabe> If you don't want to be pinged remove your name from this file: https://github.com/coreos/fedora-coreos-tracker/blob/main/meeting-people.txt

17:47 <bgilbert> marmijo[m]: yeah, makes sense. :-(

17:48 <marmijo[m]> bgilbert: I saw your message in Slack. Thank you!

17:48 <bgilbert> +1

17:51 <bgilbert> https://github.com/coreos/fedora-coreos-config/pull/2501

18:18 <bgilbert> https://github.com/coreos/ignition/pull/1673

21:18 <dustymabe> ravanelli: let me know when to do another round of reviews