01:22
jdirte has joined #fedora-coreos
01:23
jdirte has left #fedora-coreos [#fedora-coreos]
01:25
Betal has joined #fedora-coreos
02:49
plarsen has quit [Remote host closed the connection]
03:13
jlebon has quit [Ping timeout: 248 seconds]
04:20
nb has quit [Ping timeout: 268 seconds]
04:33
gursewak has quit [Ping timeout: 265 seconds]
04:36
travisghansen has quit [Ping timeout: 250 seconds]
04:39
travisghansen has joined #fedora-coreos
05:01
paragan has joined #fedora-coreos
06:09
smitterl has joined #fedora-coreos
06:09
sentenza has quit [Remote host closed the connection]
06:14
Betal has quit [Quit: WeeChat 3.8]
06:57
saschagrunert has joined #fedora-coreos
07:16
jcajka has joined #fedora-coreos
07:57
apiaseck has joined #fedora-coreos
08:26
nb has joined #fedora-coreos
09:27
jpn has joined #fedora-coreos
09:31
darknao has quit [Quit: WeeChat 3.6]
09:31
darknao has joined #fedora-coreos
10:19
jpn has quit [Ping timeout: 240 seconds]
10:22
jpn has joined #fedora-coreos
10:28
jpn has quit [Ping timeout: 268 seconds]
11:20
jpn has joined #fedora-coreos
11:25
vgoyal has joined #fedora-coreos
12:24
dustymabe has quit [Quit: WeeChat 3.6]
12:27
jlebon has joined #fedora-coreos
13:03
smitterl has quit [Ping timeout: 268 seconds]
13:09
nalind has joined #fedora-coreos
13:21
mheon has joined #fedora-coreos
13:27
dustymabe has joined #fedora-coreos
13:29
<
dustymabe >
marmijo[m]: mind holding off on progressing the FCOS releases while I investigate something
13:39
<
marmijo[m] >
dustymabe: yeah I’ll hold off for a while. Let me know when I can continue
13:46
plarsen has joined #fedora-coreos
14:00
jcajka has quit [Quit: Leaving]
14:28
cyberpear has joined #fedora-coreos
14:42
smitterl has joined #fedora-coreos
14:53
gursewak has joined #fedora-coreos
14:54
Betal has joined #fedora-coreos
15:00
smitterl has quit [Remote host closed the connection]
15:04
saschagrunert has quit [Remote host closed the connection]
15:39
ravanell_ has joined #fedora-coreos
15:41
ravanelli has quit [Ping timeout: 260 seconds]
15:52
jpn has quit [Quit: Lost terminal]
16:13
jdirte has joined #fedora-coreos
16:15
jdirte has left #fedora-coreos ["later"]
16:31
paragan has quit [Quit: Leaving]
16:49
<
dustymabe >
from what I've read I'm thinking we just go forward with our current releases 4.4.4 will be in the next round
16:58
ravanell_ has quit [Remote host closed the connection]
16:59
ravanelli has joined #fedora-coreos
17:07
<
dustymabe >
marmijo[m]: I think we can move forward with the releases (unless jlebon comes to a different conclusion than I did in ^^)
17:07
* dustymabe
goes for food
17:08
<
marmijo[m] >
dustymabe: sounds good to me
18:05
<
jlebon >
dustymabe, marmijo[m]: agreed
18:09
<
dustymabe >
jlebon: so I'm digging into this secureboot upgrade test failure
18:09
<
dustymabe >
seems like instances get to `37.20230322.2.0` and then hit a kernel panic
18:14
* dustymabe
bumps memory and re-runs in a local test
18:14
<
jlebon >
dustymabe: that's odd. yeah, worth trying for more info. though i don't know why it would require it for an upgrade but not in CI when we initially shipped this
18:15
<
jlebon >
unless old GRUB used memory less efficiently somehow?
18:15
<
dustymabe >
for #1382 I don't think we ever did anything (i think the problem just went away)
18:15
<
dustymabe >
yeah, it appears in my local test to be getting past the first upgrade now
18:17
<
dustymabe >
jlebon: maybe when there are two entries in grub it uses more memory?
18:17
<
jlebon >
ok fun. so possibly a memory bug in older grub + newer fatter kernel & initrd
18:17
<
dustymabe >
jlebon: I don't even think we can call it older grub
18:18
<
dustymabe >
in my local test it starts from `37.20221225.2.1`
18:18
<
jlebon >
ohhh, this makes me realize we didn't run the regular upgrade test in !bios
18:19
<
jlebon >
if we did, maybe we would've caught this with our 1G default
18:20
<
dustymabe >
it's weird to me that grub would take up more than 1G of memory
18:20
<
dustymabe >
it's literally the first thing that happens on the system (i.e. you'd think low resources)
18:20
<
dustymabe >
and.. what's so special about secure boot?
18:20
<
jlebon >
it could be some kind of memory layout preparation thing
18:21
<
jlebon >
dustymabe: starting from what release does it work with the default 1G?
18:22
<
dustymabe >
maybe re-define your question :)
18:22
<
dustymabe >
they all seem to fail
18:23
<
dustymabe >
i.e. they upgrade until they select `37.20230322.2.0` and then once they try to boot `37.20230322.2.0` we get the
18:23
<
dustymabe >
error: ../../grub-core/kern/mm.c:376:out of memory.
18:23
<
dustymabe >
so if we start at f35 or f36 or f37
18:23
<
jlebon >
ahh OK. I misunderstood the context then. so if I take the latest e.g. stable and do a fake upgrade in secureboot mode with 1G, it should fail?
18:24
<
dustymabe >
jlebon: I
*think* so
18:24
<
jlebon >
let me try that
18:24
<
dustymabe >
obviously it could be a specific update within 37 somewhere that introduced the problem
18:24
<
dustymabe >
but for now we can just assume all 37
18:25
<
dustymabe >
and see where it becomes not true
18:39
<
dustymabe >
jlebon: any luck?
18:40
<
jlebon >
`kola run-upgrade --qemu-firmware uefi-secure` failed for another reason. i also couldn't reproduce it manually.
18:40
<
jlebon >
mind filing an issue for this with the exact output? and maybe for now we bump memory to 1.5G for secureboot
18:41
<
dustymabe >
"i also couldn't reproduce it manually" -> it should repro easy enough if you just boot a `37.20221225.2.1` with 1G of RAM
18:42
<
dustymabe >
it comes up and then does an update - on next reboot -> kernel panic
18:46
<
jlebon >
hmm, works here
18:47
<
dustymabe >
interesting
18:47
<
dustymabe >
what happens if you run: `cosa kola run --tag extended-upgrade --build=37.20221225.2.1 --qemu-firmware=uefi-secure` ?
18:57
<
jlebon >
--- PASS: ext.config.upgrade.extended (164.62s)
18:57
<
jlebon >
ahhh interesting
18:58
<
jlebon >
`dnf update` in my pet container shows there's a grub2 update available
18:59
<
jlebon >
well no, the EFI firmware i think comes from edk2-ovmf ?
18:59
<
dustymabe >
dang -did the pipeline go down?
19:00
<
jlebon >
dustymabe: i think at this point, we should file something and track it there to not lose the details
19:00
<
ravanelli >
dustymabe: I don't think so, I did some tests with rhcos/fcos and they seem ok
19:00
<
jlebon >
not loading here either, looking
19:00
apiaseck has quit [Ping timeout: 248 seconds]
19:01
<
ravanelli >
I was only waiting for the QE test part
19:01
<
jlebon >
and... the cluster console isn't loading either
19:01
<
ravanelli >
s/I merge/I can merge
19:02
<
dustymabe >
ravanelli: i'm good with it
19:03
<
ravanelli >
jlebon: cc ^
19:05
<
dustymabe >
jlebon: at the top of the web console I see: "This cluster is updating from 4.11.26 to 4.12.9"
19:06
<
jlebon >
ravanelli: also good with it!
19:08
<
dustymabe >
marmijo[m]: just try to re-run the jobs when things come back
19:09
<
jlebon >
dustymabe: weird, i don't see that
19:09
sentenza has joined #fedora-coreos
19:09
<
dustymabe >
jlebon: it's gone now
19:09
<
dustymabe >
it was a yellow banner at the top of the console
19:10
<
marmijo[m] >
dustymabe: Will do. Things are still loading I suppose, so I'll give it a bit
19:10
<
nirik >
yeah, made the jump to 4.12.x :)
19:11
<
dustymabe >
jlebon: so.. you're telling me when you tried the `cosa kola` command I pasted you earlier it passed for you locally?
19:15
<
jlebon >
dustymabe: yup
19:16
<
dustymabe >
you can see from that the number of scenarios that hit this problem
19:16
<
dustymabe >
note that all of those are run with `--allow-rerun-success`, which means it had to fail twice in a row in order to truly fail
20:48
apiaseck has joined #fedora-coreos
20:51
travisghansen has joined #fedora-coreos
21:10
gursewak has quit [Ping timeout: 246 seconds]
21:10
gursewak has joined #fedora-coreos
21:13
apiaseck has quit [Quit: Leaving]
21:18
bgilbert has joined #fedora-coreos
21:23
plarsen has quit [Quit: NullPointerException!]
21:23
plarsen has joined #fedora-coreos
21:26
plarsen has quit [Client Quit]
21:26
plarsen has joined #fedora-coreos
21:28
gursewak has quit [Read error: Connection reset by peer]
21:30
plarsen has quit [Client Quit]
21:31
plarsen has joined #fedora-coreos
21:33
troglodito has quit [Ping timeout: 265 seconds]
21:40
troglodito has joined #fedora-coreos
21:56
nalind has quit [Quit: bye for now]
22:26
<
jlebon >
see y'all tmw!
22:27
jlebon has quit [Quit: leaving]
22:46
Guest65 has joined #fedora-coreos
22:47
Guest65 has quit [Client Quit]
22:50
vadaosman[m] has joined #fedora-coreos
23:30
gursewak has joined #fedora-coreos
23:33
vgoyal has quit [Quit: Leaving]
23:36
mheon has quit [Ping timeout: 255 seconds]