plarsen has quit [Remote host closed the connection]
jpn has quit [Ping timeout: 248 seconds]
jpn has joined #fedora-coreos
gursewak has joined #fedora-coreos
jpn has quit [Ping timeout: 248 seconds]
gursewak has quit [Ping timeout: 276 seconds]
gursewak has joined #fedora-coreos
gursewak has quit [Ping timeout: 246 seconds]
gursewak has joined #fedora-coreos
gursewak has quit [Remote host closed the connection]
gursewak has joined #fedora-coreos
ravanelli has quit [Remote host closed the connection]
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 246 seconds]
jpn has joined #fedora-coreos
vgoyal has quit [Quit: Leaving]
jpn has quit [Ping timeout: 246 seconds]
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 255 seconds]
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 252 seconds]
bgilbert_ is now known as bgilbert
<bgilbert>
lucab: are you able to do FCOS release builds on Wednesday morning? you're at the top of the standby executor list. travier[m] is out, so ravanelli would be the next fallback
bgilbert has quit [Ping timeout: 248 seconds]
jmarrero has quit [Ping timeout: 246 seconds]
Sheogorath[m] has quit [Ping timeout: 246 seconds]
Orotheyshe[m] has quit [Ping timeout: 246 seconds]
sgallagh has quit [Ping timeout: 246 seconds]
vladan[m] has quit [Ping timeout: 246 seconds]
bytehackr has joined #fedora-coreos
jmarrero has joined #fedora-coreos
Sheogorath[m] has joined #fedora-coreos
sgallagh has joined #fedora-coreos
Orotheyshe[m] has joined #fedora-coreos
vladan[m] has joined #fedora-coreos
paragan has joined #fedora-coreos
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 255 seconds]
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 255 seconds]
gursewak has quit [Ping timeout: 276 seconds]
bytehackr has quit [Remote host closed the connection]
bytehackr has joined #fedora-coreos
paragan has quit [Ping timeout: 252 seconds]
paragan has joined #fedora-coreos
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 252 seconds]
jcajka has joined #fedora-coreos
saschagrunert has joined #fedora-coreos
saschagrunert has quit [Remote host closed the connection]
saschagrunert has joined #fedora-coreos
c4rt0 has joined #fedora-coreos
paragan has quit [Ping timeout: 252 seconds]
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 252 seconds]
paragan has joined #fedora-coreos
paragan has quit [Ping timeout: 248 seconds]
paragan has joined #fedora-coreos
Betal has quit [Quit: WeeChat 3.7.1]
jpn has joined #fedora-coreos
saschagrunert has quit [Read error: Connection reset by peer]
sgrunert has joined #fedora-coreos
jpn has quit [Ping timeout: 252 seconds]
c4rt0 has quit [Remote host closed the connection]
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 248 seconds]
c4rt0 has joined #fedora-coreos
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 252 seconds]
fifofonix has joined #fedora-coreos
fifofonix has quit [Client Quit]
ravanelli has joined #fedora-coreos
mhayden has joined #fedora-coreos
sgrunert has quit [Read error: Connection reset by peer]
saschagrunert has joined #fedora-coreos
crobinso has joined #fedora-coreos
vgoyal has joined #fedora-coreos
ravanelli has quit [Read error: Connection reset by peer]
ravanelli has joined #fedora-coreos
jpn has joined #fedora-coreos
jpn has quit [Ping timeout: 248 seconds]
bgilbert has joined #fedora-coreos
mheon has joined #fedora-coreos
<dustymabe>
bgilbert: jlebon: I just looked at some of the recent builds of openssl - seems like it takes about an hour to build the RPM - so we'll have some time to plan before we'll even have a build
plarsen has joined #fedora-coreos
<bgilbert>
+1, good thought
<bgilbert>
assuming this CVE will be newsworthy, should we think about a coreos-status post too?
<spresti[m]>
bgilbert: I think we should plan for that. If its impactful, people will be looking for confirmation that a product they are using is either unaffected or patched
<spresti[m]>
What medium do we use for posting those type of things?
<dustymabe>
jlebon: I disabled auto merge on https://github.com/coreos/coreos-assembler/pull/3153 because of stability for today's anticipated releases, but maybe a better instead of freezing on COSA is to just make a quay tag and then we don't need to freeze
<jlebon>
dustymabe: i'd say so. that way we can use it for the second set of releases too to be safe and not block cosa PRs for multiple days. but maybe a branch instead so we can get any cosa fixes in for the releases?
<ravanelli>
dustymabe: jlebon Do you guys need a hand in something else?
<dustymabe>
jlebon: I created that tag - we can branch off of it if we need
sgrunert has joined #fedora-coreos
<jlebon>
dustymabe: ack
saschagrunert has quit [Read error: Connection reset by peer]
<bgilbert>
ravanelli: unless we hear back from lucab, you're the next available person on the FCOS standby rotation. will you have time tomorrow to do FCOS releases?
<bgilbert>
ravanelli: if you're too busy with the pipeline work, we can continue down the standby list
<ravanelli>
bgilbert: tomorrow is holiday here in Brazil âšī¸. Can we do it today?
<bgilbert>
ravanelli: we have another set of releases to do first (spresti[m] is handling those) and the timing is unknown
<dustymabe>
Pre-announcements of CVE-2022-3602 described this issue as CRITICAL. Further analysis based on some of the mitigating factors described above have led this to be downgraded to HIGH.
<bgilbert>
arbitrary number of bytes containing the '.' character...........
paragan has quit [Quit: Leaving]
<dustymabe>
ravanelli: this is the code I had started on this morning (breaking out the cloud replication into a `clouds_replicate()` function: https://paste.centos.org/view/aa7eb040
<jlebon>
doesn't seem to be on the access.redhat.com yet
<dustymabe>
ravanelli: it's just a stencil (nothing works yet) and a bunch of copied code in there which probably doesn't work yet
<ravanelli>
dustymabe: đ
<dustymabe>
bgilbert: jlebon: thoughts on strategy?
<bgilbert>
I'm still looking at one thing, which is "requires either ... or for the application to continue certificate verification despite failure to construct a path to a trusted issuer"
<bgilbert>
we use curl --insecure during rootfs fetch and I want to verify that that doesn't imply doing ^
<jlebon>
FWIW (and it might not be worth much), but naively grepping the openssl build logs in fedora at least shows it's compiled with `-fstack-protector-strong`
<bgilbert>
+1
<jlebon>
i think we should wait for the RH page to confirm impact
<dustymabe>
after we get our strategy set i'm going to grab lunch (while the builds are running)
<bgilbert>
I don't think we're in case 2
<bgilbert>
I suppose there's an argument for case 0 (just do regular promotions and let it land in stable in two weeks), but
<bgilbert>
it's still a high-profile vuln, and it's possible that it'll get worse once there are more eyes on it?
<dustymabe>
yeah
<jlebon>
agree
<dustymabe>
Option 1b then?
<spresti[m]>
+1
<bgilbert>
rereading, one sec
<bgilbert>
this sorta assumes no malicious certs will get through a CA, which is probably optimistic
<bgilbert>
but it does raise the barrier to attack
<jlebon>
bgilbert: when you say `curl --insecure` is safe, you mean it skips all the cert verification bits?
<bgilbert>
OpenSSL doesn't provide an easy way to skip verification, it just disables caring about the result
<bgilbert>
the key is that there isn't a verify callback that ignores the result of the standard verifier and pokes at the cert anyway
<bgilbert>
AIUI
<jlebon>
gotcha
<jlebon>
i don't *think* there's anything else in our stack that would want to look at a cert directly even after it fails verification
<bgilbert>
and most of our stack isn't OpenSSL
<jlebon>
so assuming we can rule out the "or for the application to continue certificate verification", it leaves only "requires a CA to have signed the malicious certificate"
<spresti[m]>
Ok I am going to start going with the fcos releases 1b.
<bgilbert>
spresti[m]: +1
<bgilbert>
so this'll be `next` and `testing` releases without promotions (no `ok-to-promote` label), and we'll need to backport the lockfile updates to the `next` and `testing` branches
<bgilbert>
we don't have builds yet, so there's nothing to be done
<bgilbert>
I guess actually the fast-track workflow can be used to land the fast-tracks directly in those branches
<bgilbert>
spresti[m]: ^
mnguyen has quit [Remote host closed the connection]
<bgilbert>
testing-devel is the default, so the workflow skips the prefix
<bgilbert>
s/default/usual case/
<spresti[m]>
Welp lol I guess today I noticed it.
* dustymabe
back
* dustymabe
reads scrollback
<dustymabe>
spresti[m]: bgilbert: FYI - I disabled the `build` job earlier today in jenkins - when you're ready to start the builds just re-enable the job and then kick off the builds
bytehackr has quit [Ping timeout: 255 seconds]
bytehackr has joined #fedora-coreos
jpn has joined #fedora-coreos
<spresti[m]>
+1 thank you
<dustymabe>
spresti[m]: limit the runs to just the `next` and `testing` releases (i.e. don't start more than two builds). you can re-disable the job after you've started those two
<spresti[m]>
kk
bytehackr has quit [Ping timeout: 248 seconds]
<spresti[m]>
still waiting on ci to build, once I merge I will be on those steps
<dustymabe>
i pre-seeded the packages in `coreos-pool` so there will be no koji tagger delay. IOW - there should be no need to delay any when you merge the PR and start the build
<bgilbert>
dustymabe: disabled it to preserve capacity?
<dustymabe>
bgilbert: right - if we didn't then the `testing-devel` and `next-devel` would auto kick off when the PRs merged
<bgilbert>
lucab: +1. we shouldn't start any builds yet though; will keep you updated
<jlebon>
lucab++
<lucab>
eh, bad wording, by "take" I meant "self-assign" really
<bgilbert>
+1
<bgilbert>
both Bodhi updates have passed the karma gate and are waiting on test gating
<bgilbert>
dustymabe: should we do the extra steps to update containers before the fix goes stable, or just wait?
<dustymabe>
bgilbert: oh I see - IOW should we make extra commits to our source control to suck in the "not yet stable" package directly from koji, or should we just wait til it hits bodhi stable (and the normal repos) and make sure our containers are rebuilt?
<bgilbert>
right
<dustymabe>
i'm OK either way, I trust your judgement better than my own here
<bgilbert>
meh. I don't know the HTTPS request surface of those containers, but I think we probably can afford a few hours
<bgilbert>
do we still think we should do a coreos-status post?
<dustymabe>
`a few hours` is probably generous - we'll probably need to check with releng to see if they are doing any special bohdi wheel turning for this
<dustymabe>
bodhi*
jpn has quit [Ping timeout: 255 seconds]
<bgilbert>
ah, fair
<jlebon>
i think it'd be good to preempt questions about it
<dustymabe>
looks like nirik was mentioning it in `#fedora-releng
<jlebon>
(re. coreos-status post)
<jlebon>
dustymabe: is cosa missing from that hackmd on purpose?
<dustymabe>
and I see: openssl-libs aarch64 1:3.0.5-2.fc36 f36-coreos-continuous
<dustymabe>
đ
Betal has quit [Ping timeout: 252 seconds]
Betal has joined #fedora-coreos
<bgilbert>
do we have good predictions for tomorrow's version numbers?
<dustymabe>
if we don't run `bump-lockfile` today then they should be predictable
<dustymabe>
i'll disable that job
<bgilbert>
oh nice
<jlebon>
it's not 100% though because an aborted build might've still reserved a build ID
<bgilbert>
jlebon: that only affects the A component, right?
<dustymabe>
well - the X and Y will be predictable in `X.Y.Z` format - Z should be `0`, but might need to be more than 0
<jlebon>
right
<dustymabe>
sorry I forgot a field
<jlebon>
:)
<dustymabe>
X.Y.Z.A
<dustymabe>
A it's guaranteed
<dustymabe>
isn't*
<dustymabe>
but if our luck is good it will be `0`
bgilbert_ has joined #fedora-coreos
<bgilbert_>
+1
<bgilbert_>
dustymabe: could you do the version math for me?
<dustymabe>
haha - /me is still running f35 on my desktop/laptop (I do major updates once a year) - my openssl is unaffected đ! openssl-1.1.1q-1.fc35.x86_64
bgilbert has quit [Ping timeout: 252 seconds]
bgilbert_ is now known as bgilbert
<bgilbert>
dustymabe: my connection was flaking, did you get my question about version math?
<dustymabe>
z: 20 (mapped from stream testing-devel)
<dustymabe>
n: 0 (previous version 36.20221031.dev.0 does not match scheme)
<dustymabe>
36.20221030.20.0
<bgilbert>
jlebon: I don't think it's parsed as tz-aware
<dustymabe>
s/20/2/ and I think you have the answer for `testing`
<dustymabe>
for `next-devel` I see:
<dustymabe>
x: 37 (from manifest)
<dustymabe>
y: 20221031 (from lockfile)
<dustymabe>
z: 10 (mapped from stream next-devel)
<dustymabe>
n: 0 (previous version 36.20221031.dev.0 does not match scheme)
<dustymabe>
37.20221031.10.0
<jlebon>
bgilbert: email LGTM
<bgilbert>
dustymabe: +1, thanks
<jlebon>
bgilbert: ack, i'll have to look at this code again at some point
<bgilbert>
dustymabe: if I can get an ack on the email, I'll send
<dustymabe>
i'm reading it now
<jlebon>
bgilbert: let's wait until we actually have a streams PR up?
<jlebon>
before that, we might have to rebuild
<dustymabe>
i'm no so much worried about a `streams` PR as I am these version numbers. We still don't know the A number (these builds could fail) and we won't know that information for `stable` until much later than we want to send the release announcement I think
<bgilbert>
I think if X.Y.Z.A, A=0 becomes unobtainium, and A=1 exists, there shouldn't be any confusion
<bgilbert>
diverging date stamps would be a bit more confusing
<dustymabe>
yeah
<dustymabe>
it's not ideal, but probably harmless
<dustymabe>
jlebon: WDYT?
<dustymabe>
bgilbert: wow.. didn't know about [[1]] in hackmd - is that a generic markdown thing?
<bgilbert>
[1] is generic markdown; [[1]] draws the brackets around it
<dustymabe>
nice.
<jlebon>
dustymabe: agreed. i think it's reasonable to assume that in the event users see a larger A, they'll understand it includes the fix
<dustymabe>
i guess that makes it a little easier to cross post the same content in the discussion forum
<dustymabe>
bgilbert: ack on the HACKMD content
<dustymabe>
jlebon: did you still have concerns over timing of announcement?
<jlebon>
my concern was the same (version numbers for next/testing), but by the same argument it's harmless
<bgilbert>
dustymabe: I end up cleaning it up for plain-text anyway, but that syntax gets it closer
<bgilbert>
I've rewritten the paragraph right before the code block since the private draft
<dustymabe>
spresti[m]: looks like the openstack testing is our current bottleneck
ravanelli has joined #fedora-coreos
bgilbert has quit [Remote host closed the connection]
bgilbert_ has joined #fedora-coreos
ravanelli has quit [Ping timeout: 255 seconds]
bgilbert_ is now known as bgilbert
<dustymabe>
bgilbert: the only thing about the annoucement is that the steps for "manually run the following commands on affected machines" seems a bit complex
<bgilbert>
I didn't hear back so I already sent it :-/
<dustymabe>
oh haha
<dustymabe>
nvm
<bgilbert>
and yes, they are, but there weren't great ideas for minimizing further, and also hopefully not very many people will need to run them
<jlebon>
one alternative was `rpm-ostree cleanup -pr` but that has the obvious side-effect of losing your rollback
<dustymabe>
spresti[m]: bgilbert: I have to go take over on kid duty - will check back in here in a bit - i think the releases are in good shape - just waiting on the last few tests to roll in
<bgilbert>
+1
dustymabe has quit [Quit: WeeChat 3.5]
dustymabe has joined #fedora-coreos
<dustymabe>
bgilbert: when were you planning on merging the GRUB CVE PRs ?
<bgilbert>
dustymabe: for the cosa one I was waiting on good FCOS builds for the OpenSSL CVE, but now I'm waiting on the RHCOS job
<bgilbert>
it's been running(?) for two hours with no output
<bgilbert>
and we need the cosa one merged for the f-c-c one to pass CI
djinni` has quit [Quit: Leaving]
<bgilbert>
but ASAP aside from that
<dustymabe>
ok cool
<dustymabe>
when all that merges can you re-enable the build job in jenkins and start `testing-devel` and `next-devel` builds?
<dustymabe>
when the PR merges it should kick off automatically
<bgilbert>
cool cool
<bgilbert>
re RHCOS, if push comes to shove, should I shove?
<dustymabe>
yeah, I assume
<bgilbert>
+1
<bgilbert>
spresti[m]: release job finished
<dustymabe>
re: the RHCOS job - if you look at the linnk there is a link to a openshift cluster - i think you can log in with Red Hat SSO and see more about progress of the testing
<bgilbert>
not finding a login link, but job history has runs taking north of 2h45m (!) so we're still within the envelope
<spresti[m]>
bgilbert: Yep, I am at the stage of running the roll out
<spresti[m]>
s//-/
<spresti[m]>
What roll-out time-frame did we want to go? 24h?
<bgilbert>
no longer than 24
<bgilbert>
I don't think we really need shorter. it's valid for two rollouts to overlap
<bgilbert>
the main thing is that we get bake time with _some_ of the nodes
<spresti[m]>
Ok, I will set it for 24 then. Thank you.