13:35 <dustymabe> saqali: jlebon: here are the results from the "after fresh boot" prune tests: https://paste.centos.org/view/66388e6b

13:35 <dustymabe> so with saqali PR it's at 4m and without it's 11m

13:36 <dustymabe> I suspect a lot of this is highly dependent on the current load of the netapp storage server

13:38 <jlebon> dustymabe: nice. thanks for testing that

13:41 <dustymabe> jlebon: no

13:41 <dustymabe> np, rather :)

13:42 <dustymabe> jlebon: the kexec-tools codebase has a function for checking if a system is ostree.. they grep /proc/cmdline for ostree.. would it be better to check for /run/ostree-booted ?

13:46 <jlebon> usually yes, but in what context does it run?

13:47 <dustymabe> I have no idea, I assume just in the root of the system (no namespacing or containerization)

13:47 <dustymabe> I guess I could test it

13:48 <jlebon> then yes :)

14:45 <dustymabe> hey adamw - your proposal at https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/JK4TZIRE3QP32VIT5BOXTIPMKRATTAAM/ is interesting.. we are also running tests against rawhide against FCOS, but obviously after the RPMs hit the repos. It might be worth us integrating further to hook in before RPMs hit the repo so we could possibly gate as well. Or at least give

14:45 <dustymabe> you more information about the failures you are seeing.

14:45 <dustymabe> I'd also like to discuss any further QA concerns that still exist in regards to making FCOS a top level edition

15:23 <adamw> i think i had a call with someone about that a few weeks ago?

15:25 <adamw> as for gating, gating a rawhide update on some test is not technically difficult. it can be done on a per-package basis in the package's gating.yml file or on a distro-wide basis in the greenwave config stored in infra ansible. the test result must be reported to resultsdb in a way that greenwave understands it to be associated with either a package in the update or the update itself (I can provide more info on the technical details there if

15:25 <adamw> desired), and obviously it must run fairly promptly after the package is built or the update created.

15:29 <dustymabe> can you tell me more about the `distro-wide` option?

15:30 <dustymabe> adamw: i.e. a package has passed individual package tests and now gets submitted as candidate for rawhide inclusion and there is a set of tests that run on the current rawhide package set?

16:09 <dustymabe> jlebon: for podman remote - WDYT about just uploading directly from the remote builders to things like s3 and clouds (for cloud images)? should be simple enough to copy the creds up front into the remote session and they will be bound to the lifetime of the container (or we could remove as soon as the upload is done)

16:11 <dustymabe> I had previously decided against something like this, because I was thinking we'd need to creds on the remote permanently, but if we can bind it to the life of the container then I think that might be OK

16:14 <jlebon> dustymabe: how hard would it be to switch to dynamically provisioned builders as part of this?

16:19 <dustymabe> I think I'd prefer not to block on that.. still a few kinks to work out there:

16:19 <dustymabe> 1. multi-arch cosa image in quay.io (hopefully coming soon)

16:19 <dustymabe> 2. support for provisioning ibmcloud instance in mantle

16:23 <jlebon> ack ok. anyway, i think that sounds reasonable yeah

16:23 <dustymabe> cool

16:24 <dustymabe> i'm progressing on tweaks to the pipeline that will make the build and build-arch jobs much more similar

16:24 <jlebon> awesome

16:24 <dustymabe> then I'll reverse engineer this back into cosa and see if I can get it all wired up right

16:25 <jlebon> +1

18:05 <adamw> dustymabe: what i mean by 'distro-wide' is you can say that all packages in the distro (or all critpath packages) must pass a test

18:05 <adamw> dustymabe: this is the config file for greenwave that ultimately controls all gating: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/greenwave/templates/fedora.yaml

18:06 <adamw> the rule `kojibuild_bodhipush_remoterule` is the rule that lets individual packages add gating requirements in their `gating.yml` files

18:06 <adamw> the rule `bodhiupdate_bodhipush_openqa` requires that all critpath packages pass a set of openqa tests

18:09 <adamw> at present, all gating is done through bodhi and conceptually, tests have to relate to a bodhi update somehow. the rule `compose_sync_requiredtests` is/was ultimately intended to be used for *compose* gating - the idea being that we would not sync a rawhide compose to mirrors unless it passed all those tests - but that's never been implemented in practice

18:16 <bgilbert> automated checklist PRs incoming

18:37 <bgilbert> I've already stared at the diffs a bunch, so probably some folks who aren't me should review

19:24 <walters> bgilbert: more info?

19:25 <bgilbert> https://github.com/coreos/repo-templates

19:25 <bgilbert> walters: for now, no semantic changes to the checklists are intended; they're just fixing up the existing desynchronization

19:25 <bgilbert> but this will obviously make it easier to make improvements

19:26 <bgilbert> walters: a couple notes for you in particular: 1) bootupd has diverged enough that I left it alone for now, and 2) this obviously affects https://github.com/coreos/coreos-installer/pull/894

19:26 <walters> ah wow

19:27 <bgilbert> walters: re 894, if you're up for it, it'd be great if you could PR the Cargo.toml changes to the other Rust repos that produce a vendor tarball

19:27 <bgilbert> and then we can update the checklist template to use vendor-filterer everywhere

19:28 <bgilbert> looks like afterburn, ssh-key-dir, zincati

19:30 <walters> I was thinking we'd land it in a few repos, validate it works for real (among other issues, CI and usual dev workflow doesn't cover actually building from the vendor sources, which is obviously an enormous gap), then move it everywhere

19:30 <bgilbert> fair

19:30 <bgilbert> in that case, we can conditionalize it in the templates repo

19:38 <walters> let's track/discuss in https://github.com/coreos/fedora-coreos-tracker/issues/1240

19:40 <bgilbert> +1

19:41 <bgilbert> walters: want me to break out the coreos-installer PR, or do you want to do it?

19:41 <bgilbert> walters: aside from fixing the implicit merge conflict, it's ready to go as far as I'm concerned

19:41 <bgilbert> (s390x CI is broken again, sigh)

19:45 <walters> I don't know about you mean by breaking out the PR

19:46 <bgilbert> walters: the release checklist template in coreos/coreos-installer is now a generated file, and the source of truth is in coreos/repo-templates

19:47 <bgilbert> so your changes to the checklist will need to be migrated over

19:47 <bgilbert> I can do it, but feel free if you want to play with the new system a bit

19:47 <walters> ah right, that makes sense now that I put two and two together 😄

19:48 <bgilbert> (to make the diff clearer, it'd be good to land https://github.com/coreos/coreos-installer/pull/895 first)

19:49 <bgilbert> walters: lmk how you want to proceed

19:50 <walters> if you don't mind picking up that ball a bit that'd be cool

19:51 <bgilbert> np. mind reviewing 895?

19:57 <dustymabe> jlebon: did we ever get anything up for that cannot allocate memory issue?

19:57 <dustymabe> the 3G bump stalled right?

19:57 <dustymabe> i'm hitting it constantly in https://github.com/coreos/coreos-assembler/pull/2939#issuecomment-1165551459

20:00 <jlebon> dustymabe: should have something pushed before eod. well, i already have something, just trying to see if i can improve it :)

20:04 <dustymabe> +1 thanks