00:02 <adamw> Looks good now

14:29 <jlebon> dustymabe: nice work on https://github.com/fedora-silverblue/issue-tracker/issues/405! had you figured out what happened there?

14:30 <dustymabe> jlebon: not exactly - I think I need some help :)

14:32 <dustymabe> jlebon: I sent you a full log from the pruner run

14:33 <dustymabe> I cross referenced the pruner run logs with the emails that come into the releng-cron list

14:34 <jlebon> nice ok

14:35 <dustymabe> if you look at this page of the archives: https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/2023/1/?page=8

14:37 <dustymabe> you'll see that the failure for ostree summary -u started happening with the 00:46 UTC email

14:37 <dustymabe> which was 18:46 on the 28th

14:38 <dustymabe> 00:46:52 specifically

14:38 <dustymabe> the 00:45:12 email worked fine

14:39 <dustymabe> if you cross reference that with the pruner logs it looks like this operation is the one that caused the summary update to stop working (implying commits were deleted that shouldn't have been deleted):

14:40 <dustymabe> 2023-01-29 00:45:29,168 INFO fedora-ostree-pruner - Pruning the fedora/35/x86_64/testing/kinoite ref in repo /mnt/koji/ostree/repo to time:90

14:40 <dustymabe> 2023-01-29 00:45:29,168 INFO fedora-ostree-pruner - Running command: ['ostree', 'prune', '--repo', '/mnt/koji/ostree/repo', '--commit-only', '--only-branch', 'fedora/35/x86_64/testing/kinoite', '--refs-only', '--keep-younger-than=90 days ago']

14:41 <dustymabe> and if you look at the eventual summary -u failure from the pruner - it's complaining about No such metadata object 619554b37ce12caccb9425779aca709a40fe0050d0a74fc692e34b0052a5c9f0.commit

14:41 <dustymabe> and that one was related to fedora/37/aarch64/kinoite

14:46 <jlebon> want to switch to high bandwidth?

14:47 <dustymabe> yeah give me a minute

20:06 <dustymabe> aaradhak[m]: any luck reproducing the chrony generator rawhide issue on AWS?

20:10 <aaradhak[m]> hey dustymabe just pinged you on slack regarding the same.

20:12 <aaradhak[m]> in order to reproduce the issue locally, I downloaded the 38.20230127.91.0 rawhide aarch 64 version and planning to run using cosa.

20:12 <aaradhak[m]> for instance:

20:12 <aaradhak[m]> cosa kola run --aws-credentials-file /var/home/aaradhak/.aws/credentials --qemu=fedora-coreos-38.20230127.91.0-qemu.aarch64.qcow2 ext.config.chrony.dhcp-propagation --arch=aarch64 -p=aws

20:12 <aaradhak[m]> had a doubt if this is the approach to reproduce ?

20:19 <dustymabe> so you're running in AWS. you don't need the qemu image - try with something more like:

20:19 <dustymabe> cosa kola spawn -p=aws --aws-credentials-file=/var/home/aaradhak/.aws/credentials --arch=aarch64 --aws-region=us-east-1 --aws-ami=ami-015ccd5711aed276d

20:19 <dustymabe> darn - use this instead

20:20 <dustymabe> cosa kola spawn -p=aws --aws-credentials-file=/var/home/aaradhak/.aws/credentials --arch=x86_64 --aws-region=us-east-1 --aws-ami=ami-015ccd5711aed276d

20:55 <aaradhak[m]> Hitting this error with the cosa kola spawn. the spawning instance failed - an error in resolving security group.... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/98acb60484dc3d0a35894a7faa3d721c1ef7ce27>)

20:55 <aaradhak[m]> * Hitting this error with the cosa kola spawn. the spawning instance failed - an error in resolving security group.... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/e55770c4a8c91a26fbb1af48efdfef7086cfd082>)

20:55 <aaradhak[m]> * Hitting this error with the cosa kola spawn. the spawning instance failed - an error in resolving security group.... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/a2147698df5b9b0ef2b1d66f4e7ccc2480a6322b>)

20:57 <dustymabe> aaradhak[m]: this looks like a bad assumption in https://github.com/coreos/coreos-assembler/pull/2880

21:11 <dustymabe> i'm just trying to figure out what assumption is bad.. cc @bgilbert since he generally has a good hold on networking fundamentals

21:12 <dustymabe> basically aaradhak[m] is running kola in a new AWS account and it's trying to create the kola networking resources (vpc sg subnets, etc) and failing

21:15 <dustymabe> the complain from AWS is `The IPv6 CIDR '2600:1f18:edb:e906::/64' is invalid`

21:16 <dustymabe> aaradhak[m]: if you run the command again, it continues to fail in the same way?

21:18 <aaradhak[m]> yes, it is hitting the same error on repeated runs

21:20 <dustymabe> aaradhak[m]: what does this show for you? AWS_CONFIG_FILE=/srv/creds aws --region=us-east-1 ec2 describe-availability-zones | jq .AvailabilityZones[].ZoneName

21:20 <dustymabe> the `6` in your failure message makes me think for some reason you are seeing more availability zones than I am

21:22 <aaradhak[m]> this is the output for Availability Zones:... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/ae725d174de10ae95416d1d2e56aa3ee0778e53f>)

21:23 <dustymabe> holy batman

21:24 <dustymabe> I only see the first 6

21:25 <dustymabe> https://aws.amazon.com/about-aws/global-infrastructure/localzones/locations

21:28 <dustymabe> i'm guessing the real problem isn't that the cidr is somehow invalid, but more so that one of the local zones doesn't support ipv6

21:54 <dustymabe> aaradhak[m]: can you test https://github.com/coreos/coreos-assembler/pull/3339

21:57 <aaradhak[m]> sure, will test this patch and get back.

23:33 <aaradhak[m]> dustymabe: the [patch](https://github.com/coreos/coreos-assembler/pull/3339) is working.... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/a2cb6a191ff2bb6c9ec8fdd4acbd3d9a3f528b85>)