<spresti[m]>
Hmm, I wonder; It seems like two of the builds exited on the same error, [next,test] "Importing failed: Command '['ostree', '--repo=/mnt/koji/compose/ostree/repo', 'pull-local', '/tmp/tmppb75h0j5', '607ca27fa44a4b074c33bcc4979bf476725dd513a80c530ab1b45be332f4dfdd']"
<dustymabe>
spresti[m]: looking closer now
<spresti[m]>
kk
c4rt0 has quit [Remote host closed the connection]
<dustymabe>
ok it appears to me that the ostree-importer might not be in working order
<dustymabe>
investigating that now
c4rt0_ has joined #fedora-coreos
rsalveti has joined #fedora-coreos
<dustymabe>
it looks like we're almost out of free space on that volume and the ostree importer is failing because of that
nalind has joined #fedora-coreos
<dustymabe>
ok I dropped the free space percent check to 1% and will work to run a prune operation in the background
<dustymabe>
for now we should be unblocked I think
<dustymabe>
jlebon: around? I'd like to discuss implications of the failures if you don't mind
<spresti[m]>
thank you dustymabe !
<dustymabe>
spresti[m]: let's hold off on new builds for now
<dustymabe>
still need to discuss some things with jlebon - and we still don't know exactly why the `stable` build failed
<spresti[m]>
Ah sorry.
<jlebon>
dustymabe: here now
<jlebon>
ouch, the volume on which the compose repo is hosted is running out of space?
<dustymabe>
jlebon: yeah, I never got back arond to running the ostree pruner consistently so it's expected to eventually hit the problem
<dustymabe>
that's probably a gap I should close this week
<dustymabe>
jlebon: I think the testing and next builds are salvagable, though
<dustymabe>
since the ostree import is like the last thing that runs
<dustymabe>
and the release job will run it again to import into the prod repo
<jlebon>
dustymabe: huh, i had thought it was up and operational already. is the repo on its own volume or shared with other koji things? is this affeting anyone else?
<jlebon>
i guess at least the other ostree-based variants
<dustymabe>
jlebon: yeah, theoretically the pungi composes would have this same problem
<dustymabe>
I'll run a prune operation after we get this set of FCOS releases out the door
<jlebon>
sounds good
<dustymabe>
do you agree that the `testing` and `next` build should be salvagable?
<dustymabe>
we'll have to manually run cloud tests, but that shouldn't be too hard to do
<dustymabe>
"manually run" == kick the jenkins jobs off with the right parameters
c4rt0_ has quit [Quit: Leaving]
c4rt0 has joined #fedora-coreos
<jlebon>
dustymabe: restarted CI
<jlebon>
let me check the jobs re. testing and next
<dustymabe>
IIRC when the release job runs the ostree import will happen in the prod repo then. It isn't fatal if the commit doesn't exist in the compose ostree repo already
* dustymabe
opens a PR to move the cloud tests to before the ostree import
<jlebon>
dustymabe: agreed. the importer looks like it should be able to handle that fine.
<jlebon>
i'll start the cloud tests. starting with kola-aws in case someone wants to do another
<dustymabe>
I can handle gcp
<dustymabe>
ok i'll do openstack now
<jlebon>
aws started
<jlebon>
i'll do azure
<jlebon>
done
<spresti[m]>
jlebon: Standup
paragan has quit [Quit: Leaving]
* dustymabe
brb
plarsen has joined #fedora-coreos
<spresti[m]>
Also brb
saschagrunert has quit [Remote host closed the connection]
<travier[m]>
jlebon marmijo We followed it and I made an issue in cosa for your suggestion. Not sure how practical it would be but we can try
<spresti[m]>
And to be clear even though the build failed for [next && testing] its ok to use that build and I dont need to rebuild?
<spresti[m]>
* failed for \[next &&, * && testing\] its, * to rebuild? dustymabe
poppajarv has quit [Read error: Connection reset by peer]
poppajarv has joined #fedora-coreos
<dustymabe>
spresti[m]: right. the build shows up as failed but jlebon and I determined the step was at the end and non-fatal (the release job will clean it up)
<dustymabe>
so testing and next should be good - just add a comment to the streams issues noting this
<spresti[m]>
Ok I will, finishing up another task and then I will jump back on it.