15:22 <dustymabe> marmijo[m]: check out the run times of the azure tests in https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/blue/organizations/jenkins/kola-azure/detail/kola-azure/372/pipeline/83

15:23 <dustymabe> they are completing much sooner

16:01 <marmijo[m]> dustymabe: thats good! The selinux policy was updated again recently and it looks like the builds are completing before timing out now.

16:02 <dustymabe> marmijo[m]: yeah, but let me look at something

16:06 <dustymabe> marmijo[m]: so 39.20230220.91.0 ran and passed in 1h7m (we got lucky it didn't timeout)

16:06 <dustymabe> 39.20230222.91.0 ran and passed in 22m50s

16:07 <dustymabe> now.. I did update the azure libraries backing these processes yesterday, so that could account for some change

16:07 <dustymabe> but let's focus on the package set change between 39.20230220.91.0 and 39.20230222.91.0 to see if we can pick out what we think "fixed" the problem

16:09 <dustymabe> jlebon: WDYT about adding a "releasever" denylist entry (similar to osversion, but this would make it so we don't have to add a new variable to our FCOS configs): https://github.com/coreos/coreos-assembler/pull/2934

16:11 <marmijo[m]> dustymabe: Alright, I'll start looking through the differences

16:14 <dustymabe> apiaseck: I opened https://github.com/coreos/coreos-assembler/issues/3369 to see if we can drop a test. Mind taking a look?

16:14 <apiaseck> dustymabe: Thanks - will do!

16:24 <dustymabe> jlebon: do you want to follow up with the ssh maintainers to see if they are going to join our meeting today?

16:25 <jlebon> dustymabe: ohh yes, meant to do that this morning! will shoot an email to remind them

16:28 <jlebon> dustymabe: re. releasever, it's tempting but the duplication with osversion isn't great

16:29 <dustymabe> jlebon: yeah - I just hate duplicating the information in our fcos configs (i.e. we need to update two pieces of info

16:29 <jlebon> i think we could fix that by templating the `osversion` variable

16:29 <dustymabe> true

16:30 <jlebon> but variables don't support variables right now, so it'd require rpm-ostree work

16:30 <dustymabe> does that require rpm-ostree work?

16:30 <dustymabe> yeah, don't want to create work for ourselves

16:30 <jlebon> though it requires work either way. i don't think it'd be too hard :)

16:31 <dustymabe> maybe we just redefine osversion and put it right next to releasever so it'd be hard to miss updating one if you updated the other

16:32 <jlebon> sounds fine for now, yeah

16:33 <jlebon> dustymabe: did you forget to do the ping here?

16:33 <jlebon> (for the meeting)

16:33 <dustymabe> jlebon: indeed

16:34 <dustymabe> aaradhak anthr76 davdunc dustymabe gursewak jaimelm jbrooks jcajka jdoss jlebon jmarrero lorbus miabbott nasirhm ravanelli saqali skunkerk walters

16:34 <dustymabe> FCOS community meeting in #fedora-meeting-1

16:34 <dustymabe> If you don't want to be pinged remove your name from this file: https://github.com/coreos/fedora-coreos-tracker/blob/main/meeting-people.txt

16:37 <jlebon> dustymabe: i guess another approach is to keep it as `osversion` in the denylist, and have kola look for `osversion` first, and fallback to `releasever`. it's a bit confusing, but if we doc it, it seems ok

16:49 <dustymabe> jlebon: true

17:34 <dustymabe> travier[m]: can you replace the `\ ` in the cut command in your PR to `' '`

17:39 <travier[m]> dustymabe: sure

17:40 <travier[m]> done

17:42 <dustymabe> had one more comment for you in the PR

17:47 <travier[m]> 👍️

18:43 <dustymabe> aaradhak[m]: let's screenshare and I'll show you how to debug https://github.com/coreos/fedora-coreos-tracker/issues/1429 in the pipeline

19:37 <jlebon> dustymabe: fyi i've updated https://github.com/coreos/fedora-coreos-pipeline/pull/796

20:21 <dustymabe> jlebon: reviewed

20:27 <baude> has the idea of using efi variables for ignition been kicked around here?

20:28 <dustymabe> baude: not sure, but will note EFI isn't supported on 2/4 arches we operate on

20:28 <baude> yeah, this is for a "new" platform

20:29 <baude> trying to figure out how to weasel the ignition file in

20:44 <apollo13[m]> baude: oh that would be interesting:)

20:44 <baude> it is a thing evidently

20:46 <apollo13[m]> I have no idea how to stuff that into proxmox but I guess I can figure it out. 😉

21:01 <jlebon> dustymabe: updated!

21:01 <dustymabe> marmijo[m]: were you able to figure out what update fixed the azure timeout issue?

21:05 <marmijo[m]> dustymabe: I see that libselinux was updated in 39.20230218.91.0, which was the first rawhide build to not timeout. It was still a long run though, so could have just gotten lucky. There was a kernel update between the two latest builds, but I'm not sure if anything else would have 'fixed' the issue.

21:08 <marmijo[m]> Container-selinux and selinux-policy were also updated in 38.20230216.91.0, but that build timed out. Every process is significantly shorter after you updated the azure libraries though, could it be a combination of that and the selinux related packages update?

21:11 <dustymabe> jlebon: replied

21:14 <dustymabe> marmijo[m]: hmm. well we do know that there are SELinux denials - do the logs from the newer runs have those denials in them still?

21:14 <dustymabe> clarification: "we do know that there are SELinux denials in the old failure case"

21:21 <marmijo[m]> Yes, there are still SELinux denials in the logs. It's interesting because the tests are passing in a reasonable amount of time meaning they're not waiting for the reboot

21:22 <dustymabe> marmijo[m]: interesting for sure

21:24 <dustymabe> marmijo[m]: if the denials still exist (and you can manually reproduce the issue by starting an instance and using the web interface to try to `reboot` and it does nothing) then we still need to open a BZ for this

21:24 <marmijo[m]> dustymabe: Sounds good. I'll test that out

21:28 <dustymabe> marmijo[m]: in the API update I use force delete - so maybe that could be the difference

21:29 <dustymabe> gursewak: any progress on https://github.com/coreos/fedora-coreos-tracker/issues/1402 ?

21:34 <gursewak> dustymabe, got a catch-up meeting with jlebon tomorrow.

21:38 <dustymabe> 👍

21:42 <jlebon> dustymabe: updated

21:47 <dustymabe> stamped

21:55 <jlebon> merged! was going to hold until tomorrow, but this way any errors that happen tonight will be ready to fix tomorrow :)