08:22 <stephan> we did a rollback, "tried rollforward" - did work a that time - but now auto update is broken: https://justpaste.it/8b87f

08:23 <stephan> ostree status

08:23 <stephan> https://justpaste.it/61v9n

08:24 <lucab> stephan: uhm, it looks like an rpm-ostree bug. Do you still have the history of the commands you ran?

08:25 <stephan> yes, one second

08:27 <stephan> https://justpaste.it/5ssh7 <-- most likely I used it wrong

08:27 <lucab> stephan: also, the full logs from zincati would help. It looks like the commit object for `36.20220605.3.0` somehow is missing, thus the upgrade fails.

08:51 <stephan> zincati log new->old from where is sopped working: https://justpaste.it/6zkam

08:53 <stephan> my experience with ostree is very limited

09:04 <lucab> I'm not sure how the system ended up in that situation, but I think you can temporarily `systemctl stop zincati` and then `rpm-ostree rebase fedora/x86_64/coreos/stable` to bring it back into alignment with the normal stream updates

09:05 <stephan> ok, going to try that

09:07 <stephan> that die work (did NOT fail)

09:10 <lucab> I see in your history that you tried that rebase earlier (with --bypass-driver too), it should have worked at that point. Likely whatever failure happened at that point the caused further troubles for auto-updates

09:19 <stephan> ok, that dit work, after reboot: Fedora CoreOS 36.20220605.3.0, thank's a lot

09:19 <stephan> maybe the documentation should have a hint how to correctly proceed after a manual rollback

09:20 <lucab> stephan: please keep an eye on zincati for the next upgrade. I tried to manually reproduce your situation on a fresh FCOS stable machine, but didn't manage to.

09:20 <stephan> yes keeping a close eye on that one

09:21 <lucab> stephan: yes, I think that would be good. Were you following https://docs.fedoraproject.org/en-US/fedora-coreos/auto-updates/#_manual_rollbacks ?

09:21 <stephan> yes the "Permanent rollback" path

10:02 <lucab> stephan: https://github.com/coreos/fedora-coreos-docs/pull/426

11:17 <stephan> @lucab thanks for helping, the issue looks good