travisghansen has quit [Ping timeout: 240 seconds]
bgilbert has joined #fedora-coreos
jpn has quit [Ping timeout: 246 seconds]
jpn has joined #fedora-coreos
killefiz[m] has quit [Quit: You have been kicked for being idle]
jpn has quit [Ping timeout: 250 seconds]
crobinso has quit [Ping timeout: 244 seconds]
crobinso has joined #fedora-coreos
zodbot_ has joined #fedora-coreos
zodbot has quit [Ping timeout: 260 seconds]
zodbot_ has quit [Excess Flood]
zodbot has joined #fedora-coreos
c4rt0 has quit [Quit: Leaving]
<travier[m]>
Meeting for Fedora Go/NoGO in meeting-1 now
<travier[m]>
My bad, it's "#fedora-meeting"
marmijo has joined #fedora-coreos
aaradhak has joined #fedora-coreos
<dustymabe>
travier[m]: +1
<dustymabe>
bgilbert: jlebon: lucab: ^^
<bgilbert>
+1
marmijo has quit [Quit: Client closed]
marmijo has joined #fedora-coreos
<dustymabe>
ok accepted blocker - let's meet here in 30 minutes
<bgilbert>
+1
<travier[m]>
could still potentially be waived but unlikely given the discussion
<travier[m]>
I won't be able to attend the discussion but given that Fedora is (likely) delaying, we can "just" the fix in testing on F36 and do the rebase after "as planned" (testing on GA, etc.)
<dustymabe>
👋
<dustymabe>
i'm ready when others are
<jlebon>
👋
<dustymabe>
so - f37 delayed
<anthr76[m]>
👋 does anyone know if FCOS aarch64 images are being pushed to GCP yet?
<dustymabe>
anthr76[m]: they are not, but I might have an image I created recently you can test - ping me in 30 minutes
<anthr76[m]>
Sure!
<bgilbert>
o/
<spresti[m]>
👋
<bgilbert>
looks like they're currently talking about lifting the freeze
<dustymabe>
we can proceed with rebase on f37 GA schedule then - I think the open question now is about next week's triple
<bgilbert>
yup, agreed
<jlebon>
+1
<bgilbert>
two things I noticed:
<bgilbert>
- if Red Hat Product Security is advising the FPL to respin the boot media, that implies that this is a client-side issue
<bgilbert>
recommending that customers inventory their affected machines
<bgilbert>
maybe that's boilerplate wording? but it suggests a certain urgency
<dustymabe>
seems like good practice?
<bgilbert>
so. since this is going to be a regular triple, I assume we delay for OpenSSL rather than cutting a triple + OOB release
<bgilbert>
and then the question is, do we patch stable at the same time, or do we bake the fix first
<dustymabe>
i think there is one other open question
<dustymabe>
should we promote testing at all?
<bgilbert>
testing, specifically?
<dustymabe>
i.e. instead of normal promotion should we keep testing frozen on current package set + SSL fix and ship that
<dustymabe>
then after like a day or something zip that straight to stable (so stable gets a generic package bump)
<jlebon>
if we want to fast-track it to stable, that'd be much safer yeah
<dustymabe>
then we can spin testing for generic package promotion if we want
<bgilbert>
you mean, so we don't ship a package set that hasn't been tested together
<bgilbert>
in stable
<dustymabe>
correct
<dustymabe>
the existing content in `testing` would have already baked two weeks
<dustymabe>
so baked content + new openSSL + 24h feedback period
<dustymabe>
then ship to `stable`
<jlebon>
for next week's next, should we bump content to match whatever Fedora decides for GA content?
<bgilbert>
if we're baking the fix, I agree that that's cleaner, if we're willing to do two testing builds + rollouts in a week
<dustymabe>
bgilbert: I think I'd be OK with that, but even if we didn't it wouldn't be the end of the world
<jlebon>
bgilbert: would it make sense to hold testing until the rebase?
<bgilbert>
oh yeah, the rebase :-(
<jlebon>
so we'd do just one release next week, with the openssl fix
<bgilbert>
checking the state of the Go/NoGo discussion
<dustymabe>
actually, no. that wouldn't make sense. We need something to promote to `stable` the week of the 37 rebase
<dustymabe>
well - i guess we don't have to :)
<dustymabe>
shall we make a hackmd with proposals illustrated?
<bgilbert>
dustymabe: yes, let's do that
<jlebon>
i guess it depends if everything gets unfrozen or not, because otherwise we'll have tons of packages we'd need to fast-track to avoid downgrades
<bgilbert>
looks like they're talking about slipping two weeks
<bgilbert>
yeah, we can't completely skip cutting a new `testing` because that'll cause a burp in the pipeline for `stable`
<spresti[m]>
Thank you for that, it makes it much clearer
<dustymabe>
sorry had to step away for a moment
<bgilbert>
1 vs 2 will depend on the severity/lines-of-code-changed ratio, I think
<bgilbert>
the (b) options provide more assurance but require more releases
<bgilbert>
for 1b, I think we probably ship stable and the promoted testing together
<bgilbert>
once we've completed the initial testing rollout, we don't have any further need for that initial testing release. the baking will have already been done
<dustymabe>
can we add some more details to 2a?
<jlebon>
i feel like we should rule out 1a
<bgilbert>
we can rule out 1a if we're okay with the extra work, yeah
<jlebon>
we'd be reducing our usual 2 week bake for a whole new pkgset to 24h
<dustymabe>
or possibly no extra work - see 1b.2
<bgilbert>
I don't think we can skip package updates into stable for a cycle. that'll mean that stable gets as old as 6 weeks
<bgilbert>
re 1b.2
<dustymabe>
will it :)
<bgilbert>
...yes?
<dustymabe>
so we'd promote on week -1 (content at this point is week -3)
<dustymabe>
in week 0 - no promotion
<bgilbert>
jlebon: re 1a, we're not normally cautious of shipping out-of-cycle fixes. I don't think pushing an out-of-cycle OpenSSL inherently destabilizes the packageset
<dustymabe>
week 3 - promotion
<bgilbert>
dustymabe: I don't follow the negative numbers, but: content is up to two weeks stale when it gets promoted, and then it stays in the distro for four weeks
<dustymabe>
week 0 is 37 GA
<bgilbert>
sorry, stable would get as old as 8 weeks
<bgilbert>
I'm not talking about 37, I'm talking about regular security updates
<jlebon>
bgilbert: isn't 1a implying shipping all new pkgset, not just openssl?
<dustymabe>
but we have to consider 37 - because according to our docs - we do a triple on release day
<bgilbert>
jlebon: right, and I'm saying that adding an out-of-cycle openssl doesn't automatically invalidate the two weeks of baking that the rest of testing has already had
<bgilbert>
dustymabe: 37 is being pushed two weeks, so 37 final won't shorten our lifecycle
<dustymabe>
oh wow - did they decide that?
<dustymabe>
i thought it was just smoke
* dustymabe
was still asuming it was going to be 1 week
<jlebon>
bgilbert: i think i'm misunderstanding. 1a IIUC is saying testing would be promoted with new content, and then that content will be promoted to stable 24h later. so that content will have had no baking at all
<bgilbert>
<bcotton> #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land.
<dustymabe>
jlebon: that's my understanding too
<jlebon>
unless line 8 should say promote PREVIOUS testing to stable with openssl fix
<dustymabe>
bgilbert: +1 - that changes things :)
<bgilbert>
jlebon: I understand it as saying that we follow the normal process but patch openssl into the stable promotion
<jlebon>
bgilbert: i think that's 2a :)
<dustymabe>
jlebon: yep
<dustymabe>
see my TL;DR
<bgilbert>
the difference between 1a and 2a is that 1a holds the stable release for baking time in testing
<bgilbert>
i.e. 24 hours to make sure the new openssl doesn't break anything
<dustymabe>
bgilbert: I think the confusion in 1a is you don't make it clear which `testing` you are promoting to stable
<bgilbert>
yup, sorry
<bgilbert>
the intent was that we'll need to decide 1x vs. 2x depending on how bad the vuln is
<jlebon>
right, i think you mean the previous testing, right?
<bgilbert>
yup
<bgilbert>
(whether we have the luxury of baking openssl in testing at all)
<jlebon>
bgilbert: ok, that makes more sense. added clarification in the doc
<bgilbert>
+1
bytehackr has quit [Quit: Leaving]
<dustymabe>
2b is a bit worrisome because we ship to stable with no bake
<bgilbert>
ship openssl you mean?
<bgilbert>
or other content
<dustymabe>
openssl - but in general a content set that wasn't baked together
<bgilbert>
yeah, I don't put a lot of weight on baking content together
<dustymabe>
openssl would be the only new thing
<bgilbert>
unless the OpenSSL update simultaneously updates other packages because there's a cross-pkg compat issue
<bgilbert>
in which case, yes, we should bake stuff together
<bgilbert>
unless there were cross-pkg prep changes that already went in, which I suppose is a possibility
<bgilbert>
I agree that options 2x are less ideal but we may not feel we have a choice
<dustymabe>
bgilbert: IOW if the severity is high enough?
<bgilbert>
yeah
<dustymabe>
it's good to have the options on the table
<bgilbert>
e.g. suppose it's a one-line fix for an RCE in HTTPS clients. we should probably do 2x in that case
<bgilbert>
though I guess it's a whole new OpenSSL release, so there will be unrelated changes too
<jlebon>
is option 3 a duplicate of one of the others?
<dustymabe>
looking a lot like 1b
<bgilbert>
I think it's 1b
<travier[m]>
Can we really decide without knowing how bad the vuln is for us?
<bgilbert>
travier[m]: no
<dustymabe>
travier[m]: we can narrow down the options
<bgilbert>
^
<travier[m]>
👍️
<dustymabe>
basically we can say we're going to do X if we allow ourselves some bake time and Y if we decide everyone needs it now
<dustymabe>
1b and 3 the same? anyone see any differences?
<jlebon>
yeah, i think it's a framing thing.
<jlebon>
stable wouldn't be an async release, but part of the regular cycle
<bgilbert>
yeah, I concur it's a framing thing
<jlebon>
so the only special thing we do is one async testing release
<dustymabe>
wait I'm confused
<bgilbert>
the framing on 3 seems cleaner
<jlebon>
so we'd ship stable with the openssl fix at the same time as new testing content
<dustymabe>
they are equivalent in technical details, just wording is different?
<bgilbert>
right
<dustymabe>
ha - ok :)
<bgilbert>
1b gives a list of steps, 3 observes that those steps are equivalent to a testing OOB followed by a normal promotion
<jlebon>
bgilbert: +1
<dustymabe>
yes
<dustymabe>
that's what I was going for
<bgilbert>
cool
<jlebon>
but 1b seems to imply that the second testing wouldn't be concurrent with the stable release
<bgilbert>
it need not be, but I think it should be
<dustymabe>
well I mean, it doesn't really matter
<bgilbert>
it's a little less releng work if they're concurrent
<bgilbert>
#proposed replace 1b with the contents of 3
<jlebon>
ok cool. yeah it was clearer for me written down that way :)
<dustymabe>
originally written I thought the 37 rebase was going to be the following week
<bgilbert>
ah, okay
<dustymabe>
bgilbert: WFM
<jlebon>
WFM too
<bgilbert>
#agreed replace 1b with the contents of 3
<bgilbert>
:-P
<jlebon>
haha
<jlebon>
so personally leaning towards 1b :)
<bgilbert>
I've convinced myself that baking the packageset together is useful
<dustymabe>
yeah basically 1b if things are just on fire and not a bomb about to explode
<bgilbert>
not necessarily 100% required, but still
<dustymabe>
if the time bomb is ticking then maybe 2b
<dustymabe>
haha - /me always choosing the option with 1 extra release
<jlebon>
ok, can i try to rewrite 2b?
<bgilbert>
sure
<bgilbert>
re 2a vs 2b, if we're not going to test the package set together, I think the main value of 2b is in case the testing promotion breaks something
<bgilbert>
(independent of openssl)
<bgilbert>
what's going on with the section titles?
<dustymabe>
i'm re-interpretting them into common terms
<dustymabe>
we can delete when I'm done if not useful
vgoyal has quit [Quit: Leaving]
<bgilbert>
+1
<jlebon>
have to go to a meeting, but my 2c is to keep it simple and do 1b if it's not the end of the world, otherwise 2c
<jlebon>
haha '2c' in that context as an abbreviation is not well chosen
<jlebon>
bbiab
<bgilbert>
also we haven't been talking about `next` at all
<dustymabe>
bgilbert: do you see where I'm going with the heading titles?
<bgilbert>
yup. when I was looking the option numbers didn't all line up yet
<dustymabe>
yeah that was my copy/paste error
<bgilbert>
yep, np
<dustymabe>
ok I think I'm 1b and then 2b
<dustymabe>
would prefer not to do an extra extra release
<dustymabe>
but there is an argument to be made for giving users on that stream an option for if the new package set brings in a problem for them
<bgilbert>
yeah, and also an argument for not breaking our normal processes
<bgilbert>
the `stable` promotion in 2b is a bit weird. it's probably fine but
<dustymabe>
does 2b break our normal processes?
<bgilbert>
`stable` comes from an old commit of `testing`
<dustymabe>
right
<dustymabe>
ok then 1b and 2c ?
<bgilbert>
if we're in option 2, then things are pretty bad, so leaving users the flexibility to not concurrently take a testing promotion makes some sense
<bgilbert>
also lets us do a faster rollout
<dustymabe>
yeah
<bgilbert>
+1 to 1b or 2c
<bgilbert>
we need to talk about `next` though
<dustymabe>
should we discuss `next` - I was just thinking we'd definitely ship the promotion there
<dustymabe>
but.. maybe not?
<bgilbert>
2c should probably just do two triples
<bgilbert>
makes the story easier
<dustymabe>
yeah that's fine
<dustymabe>
though - our resources aren't infinite - having them all run at the same time does slow the pipeline down some
<bgilbert>
right
<bgilbert>
we still have the option to adjust these when we see the situation
<bgilbert>
for one thing, 37 packages might not come out simultaneously with 36
<dustymabe>
ok 2c is two triples
<bgilbert>
we should prioritize stable > testing > next
<dustymabe>
what about 1b?
<dustymabe>
OOB testing and next?
saschagrunert has joined #fedora-coreos
saschagrunert has quit [Remote host closed the connection]
<dustymabe>
or OOB testing, normal next on day 0
<bgilbert>
how much change will we have had in next?
<bgilbert>
with 37 frozen
<dustymabe>
well, not much until I go in and fast-track a bunch of packages here soon because we ARE going to have another testing promotion
<bgilbert>
another next promotion?
<dustymabe>
no
<dustymabe>
basically we try to make it so that any upgrade doesn't have downgraded packages
<bgilbert>
oh, I see
<dustymabe>
in order to do that we have to keep the packages in `next` newer or equal to what's in testing
<dustymabe>
at this point a bunch of updates have made it into f36 that aren't in f37 (freeze)
<bgilbert>
in that case, we're doing the OOB testing release mainly for baking stable, not for the benefit of `testing` per se
<dustymabe>
right - and it also gives the users a "fixed (unpromoted)" rollback if the promoted content causes them issues
<dustymabe>
so there is a secondary benefit
<bgilbert>
for `testing`, it happens to. the question is, how much is that a side effect vs. a desired one
<bgilbert>
yeah
<dustymabe>
let's just say - whatever we do for testing, we'll do for `next`?
<dustymabe>
keeps it simple
<bgilbert>
yeah, that's defensible
<dustymabe>
ok 1b or 2c :)
<dustymabe>
at least that's settled
<bgilbert>
yup
<bgilbert>
cool. so now we wait until Tuesday.
* dustymabe
will go ahead and fast-track packages in `next-devel` (unrelated to this conversation, but pertinent to the "No Go")
<bgilbert>
are we doing a `next` update this week for 37 reasons?
<bgilbert>
*`next` release
<dustymabe>
we should be doing one every week
<dustymabe>
but only if content has changed
<bgilbert>
+1
<dustymabe>
hmm weird
marmijo has quit [Quit: Client closed]
<dustymabe>
the download page has `37.20221021.1.0` but the release notes page doesn't list it
<dustymabe>
oh nvm - I see it now
<dustymabe>
yeah so we did one this week already
* dustymabe
goes back to work :)
* bgilbert
waves
crobinso has quit [Remote host closed the connection]
jpn has joined #fedora-coreos
aaradhak has quit [Quit: Connection closed for inactivity]
jpn has quit [Ping timeout: 252 seconds]
<jdoss>
So I think I hit a bug in the layered updates stuff where I have version 1 that gets layered and booted, version 2 gets released and staged but not rebooted, version 3 gets released and I run rpm-ostree update and it just hangs. I started working on a GH issue with step by step instructions with a much smaller container image and it works... uigg. With the larger image it is reproducible.. I think. Should I still open the issue?
<bgilbert>
jdoss: open the issue with whatever you have, yeah
<jdoss>
walters: I should have tagged you on the above.
<jdoss>
OK on rpm-ostree repo?
<bgilbert>
that'd be a question for walters
<bgilbert>
dustymabe jlebon: couple other things on OpenSSL
<bgilbert>
technically a standby executor should be handling the OOB part, and then spresti[m] would take over for the normal promotions
<bgilbert>
at this point spresti[m] has been planning to handle the first set, so maybe it makes sense to have a standby executor handle the second set instead
<bgilbert>
also: depending on the details, we may also have container images to update
<bgilbert>
lucab is next on the standby list
<dustymabe>
whatever happens for this it's going to need to happen fast
<jdoss>
The openSSL stuff? that bad?
<dustymabe>
i don't know if spresti[m] is going to want me bugging him about it :)
<dustymabe>
bgilbert: the announcement won't happen until later in the day for lucab - would hate to put that on him
<bgilbert>
jdoss: we don't know how bad it'll be, just trying to plan
<jdoss>
fair enough. I appreciate you all.
<bgilbert>
<3
<bgilbert>
dustymabe: I've already been syncing with spresti[m] about it. I propose spresti[m] for the first set and lucab for the second set, 24h later or whenever it happens to be
<walters>
jdoss: yep definitely
<bgilbert>
the nice thing about the second set is that the _builds_ can be done async
<jdoss>
walters: I am fallocate -l 4G /usr/src/mycoolapplication.img to try and reproduce with a bigger image.
<bgilbert>
if lucab isn't up for it, travier[m] is next on the list
<dustymabe>
SGTM
<jdoss>
walters: open issue on rpm-ostree repo?
<walters>
well, it's fine to start there yeah, in practice it's probably an ostree issue but there are 3 repositories there and it doesn't matter too much, we can transfer things