13:06 <jlebon> bgilbert: not recently. do you know when it started?

16:00 <dustymabe> jlebon: did you apply any necessary changes for https://github.com/coreos/fedora-coreos-pipeline/pull/671 to the prod fcos pipeline already?

16:07 <bgilbert> jlebon: best info I have is that it worked at the beginning of last week and didn't work at the beginning of this week

16:08 <jlebon> dustymabe: ouhhh, good point. will do that after that build-cosa job is done

16:09 <dustymabe> jlebon: +1

16:09 <dustymabe> also any updates to coreos-ci that we should make based on the recent flurry of changes to the fcos pipeline (I know they share some bits IIUC)

16:12 <jlebon> nothing pressing but we should rebuild jenkins to match the recent base and plugin updates

16:14 <dustymabe> do you want to handle that bit?

16:26 <ravanelli> dustymabe: jlebon https://github.com/coreos/fedora-coreos-pipeline/pull/672 I still need to test for real, trying to upload to make sure it is working.

16:30 <dustymabe> aaradhak davdunc dustymabe gursewak jaimelm jbrooks jcajka jdoss jlebon jmarrero lorbus miabbott nasirhm ravanelli saqali skunkerk walters

16:30 <dustymabe> FCOS community meeting in #fedora-meeting-1

16:30 <dustymabe> If you don't want to be pinged remove your name from this file: https://github.com/coreos/fedora-coreos-tracker/blob/main/meeting-people.txt

16:32 <dustymabe> jlebon: looks like the COSA container build is done

16:32 <dustymabe> i'm going to spin down jenkins so nothing else starts

16:32 <dustymabe> you can spin it back up once you've made the necessary changes

16:33 <jlebon> dustymabe: ack, thanks

16:38 <jlebon> back up now and build job re-enabled

16:38 <jlebon> i've also tested that the default agent image knob is kicking in correctly

17:39 <bgilbert> jlebon: is it possible to perform operations as root inside the buildPod? sudo and su -c both don't seem to work

17:40 <jlebon> bgilbert: you want `runAsUser: 0`

17:40 <jlebon> https://github.com/coreos/coreos-ci-lib/blob/156c19554a5c33f6679210448784e67e7f1d4a6f/vars/pod.groovy#L5

17:40 <bgilbert> +1, thanks

17:52 <dustymabe> jlebon: wow - that's plumbed all the way through `cosaPod()` and works in our openshift namespaces? I thought you would need some sort of enhanced SCC for that

17:53 <walters> just as a followup from that chat, one example here...I'm pretty sure it was arithx but maybe sdemos who said in a discussion at one point something to the effect of "if you want your systems to be <reproducible/reprovisionable/something> then use ignition, if you want them to drift use ansible". But this container-native OS approach *fixes ansible*! If you commit a change to the playbook in

17:53 <walters> https://github.com/coreos/coreos-layering-examples/tree/main/ansible-firewalld then the system will reliably reprovision to that state. (I also touched on the "client side ansible" hysteresis problem in https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/ )

17:54 <jlebon> dustymabe: you do :) it won't work in the FCOS pipeline

17:54 <jlebon> dustymabe: see https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/coreos-ci/templates/securitycontextconstraints.yaml

17:57 <fifofonix> walters: re fedora proposal, i believe that typhoon is running kubelet as podman container. how does that relate to the kubernetes/kubelet example?

17:57 <jlebon> walters: definitely, the day-2 capability is really appealing

17:57 <bgilbert> walters: yeah, to be clear, I think there's value in this work. I just want to make sure we're being thoughtful about how we apply it, and not uncritically pushing everyone to simply replace all their tools

17:57 <walters> bgilbert: yeah I agree!

18:00 <walters> that said I switched out my desktop to be a custom container image (previously I'd been using this hacked up silverblue-from-fcos thing in https://github.com/cgwalters/fedora-silverblue-config ) where I'd tried to use butane to keep my laptop and desktop configs in sync, factoring out shared configs and running into butane stuff like https://github.com/coreos/butane/issues/118 ) and...now I don't need to think about butane, my shared layers

18:00 <walters> are literally just shared container images. Also Dockerfile `COPY etc etc` and storing configs in git Just Works

18:01 <walters> fifofonix: I doubt it, as the change says AFAIK kubelet doesn't support it anymore. But I haven't tried typhoon

18:04 <fifofonix> walters: i'm running a few typhoon clusters with k8s 1.24.3 and it seems to still be supported (but I don't know whether the maintainer has had to do anything special to make that happen)

18:06 <walters> IIRC the main point of contention was things like storage drivers (e.g. ceph, etc.) for whom trying to support both containerized-and-not was a huge amount of pain. It may probably be OK without those

18:10 <fifofonix> walters: i see. thanks.

18:12 <walters> I can imagine their pain... https://github.com/coreos/rpm-ostree/pull/3961 was a first foray into running rpm-ostree from inside a privileged container, and oh man was it a mess...so much code kind of assumes systemd, or if you're not in systemd that you're not operating on the booted system. My first pass at that PR deleted the running root filesystem

19:27 <jlebon> dustymabe: can you stamp https://github.com/coreos/coreos-ci/pull/46 ?

19:29 <dustymabe> jlebon: asked a question

19:30 <dustymabe> jlebon: also - is this concerning to you? https://github.com/coreos/coreos-ci-lib/blob/main/resources/com/github/coreos/job.yaml#L20

19:32 <dustymabe> I guess the "The actual container here isn't used for anything (other than staying alive for 24h" makes it less concerning, but we could probably just use a :latest tag or something for that

19:36 <jlebon> dustymabe: right, it's just to keep a container alive to use as a parent for other resources. but yeah, using :latest sounds good

19:54 <bgilbert> jlebon: the coreos-installer CI timeouts appear to be a Rust 1.64 regression

19:55 <jlebon> bgilbert: 🎉