13:28 <fifofonix> re-introduced latest next/testing to environment yesterday afternoon and recording cifs failures unfortunately. just raised an issue.

14:02 <stephanepoq> hi, I have changed a disc in my softraid, did work good. but how to "grub2-install" on the new disc? - "grub2-install: error: cannot backup `/boot/grub2/i386-pc/lsapm.mod': Read-only file system.", is ther a manual? (looks like a coreos secicfic issue)

14:46 <dustymabe> stephanepoq: you can remount the /boot fs so that it's read/write with `mount -o remount,rw /boot` but i defer to walters or jlebon for the best advice on installing the bootloader on the new disk

14:48 <dustymabe> this might be a good question for the discussion forum too - so others can see the answer

14:48 <walters> In theory bootupd should be helping with managing redundant bootloaders

14:48 <dustymabe> https://discussion.fedoraproject.org/tag/coreos

14:57 <jlebon> stephanepoq: yeah, a forum topic with more details on your setup would help. it sounds like you're not using FCOS' RAID1 sugar

15:44 <dustymabe> jlebon: wow on https://bugzilla.redhat.com/show_bug.cgi?id=2154428#c16 - I really feel like without our model of fine grained package diffs/pins/fast-tracks etc.. it would have taken A LOT longer to find the root cause of that issue. It still took us some time, but it was much easier to pinpoint things

15:47 <dustymabe> davdunc[m: any updates from your internal tracking ticket you can post to https://github.com/coreos/fedora-coreos-tracker/issues/1306 ?

15:48 <dustymabe> would also be nice to know if there's any more info on https://github.com/coreos/fedora-coreos-tracker/issues/1371

15:48 <davdunc[m> will update in the next couple of hours.

15:49 <davdunc[m> dustymabe: on another task. I'll get to it soon.

15:54 <jlebon> dustymabe: indeed, the ability to easily drive down to the specific pkg upgrade is really nice for debugging

16:27 <fifofonix> jlebon: i'm going to do that bugzilla. when you say file it against the kernel component do you mean 'kernel' or would that be 'cifs-utils'? don't understand how this all fits together.

16:28 <dustymabe> fifofonix: yeah, kernel component

16:28 <dustymabe> `kernel`

16:28 <fifofonix> +1

16:31 <jlebon> fifofonix: thanks! make sure to include the kernel versions in there

17:09 <dustymabe> fifofonix: can you try with `rawhide` and say if the kernel there helps? https://builds.coreos.fedoraproject.org/browser?stream=rawhide&arch=x86_64

17:10 <dustymabe> will help the information in the BZ - also would be good to put the full dmesg output in the BZ if you can get it somehow

17:11 <fifofonix> is full dmesg more than journal?

17:12 <dustymabe> the journal should have it in there (assuming your system doesn't crash hard and you can still get journal output)

17:13 <fifofonix> wasn't a hard crash. i put the whole journal for the boot in there.

17:13 <dustymabe> jlebon: I did it again! https://github.com/coreos/fedora-coreos-streams/pull/641

17:13 <dustymabe> need a second review on ^^ if someone can manage it: jmarrero marmijo[m] gursewak ravanell_ ^^

17:15 <marmijo[m]> dustymabe: LGTM

17:15 <dustymabe> thanks!

17:15 <jlebon> dustymabe: :)

17:21 <fifofonix> dustymabe: for rawhide is that a case of rpm-ostree deploy specific commit or is there a need to rebase to a rawhide stream or some such?

17:36 <dustymabe> fifofonix: if you can start from scratch then you grab the disk images.. if you have existing systems (preferrably ones that you don't care if something bad happens to them) then you can rebase to the rawhide stream with something like: `sudo rpm-ostree rebase fedora-compose:fedora/x86_64/coreos/rawhide`

17:37 <dustymabe> jlebon: I wanted to discuss a potential zincati suboptimal behavior with you when you get a chance

17:42 <fifofonix> dustymabe: thanks, I was using I guess what was old-syntax (without fedora-compose)...but what you gave me works. thanks.

17:44 <dustymabe> fifofonix: we use a separate ostree repo for the non prod stuff. It's behind the `fedora-compose` ostree remote.

18:10 <walters> Or you could use the container images

18:10 <fifofonix> walters: +1 ;o)

18:13 <walters> order of magnitude people who know what you mean when you say "ostree remote": 1000; order of magnitude when you talk about registries and containers: 10,000,000 at least?

18:34 <dustymabe> walters: i'm aware of all of the dynamics at play here. I don't find ^^ helpful

18:35 <dustymabe> I'll start recommending container images when we get the upcoming work out of the way

18:35 <dustymabe> hopefully we'll get started on that soon in a dedicated EPIC team

18:36 <dustymabe> which I'm anticipating will help close the gaps

18:36 <dustymabe> need one more review on https://github.com/coreos/fedora-coreos-streams/pull/642#pullrequestreview-1248254200

19:08 <dustymabe> jlebon: filed https://github.com/coreos/zincati/issues/928 so we can discuss the proper behavior there

19:59 <jlebon> dustymabe: cool, sounds good

21:07 <fifofonix> jlebon & dustymabe: cifs/kernel issue reproduced on rawhide (noted bugzilla)...but some better news on the swarm networking item...just updated the issue...