00:22 <orowith2os> <marshallford> "Hello all, I'm running into what..." <- can't you specify that the container should wait for network to be available before doing anything?

09:53 <samcday[m]> Hello! I'm trying to get stable CoreOS to boot on an old Librem 13 I have lying around. After GRUB screen I see the messages for loading kernel + initrd and then ... nothing. Screen goes blank and nothing seems to be happening.

09:53 <samcday[m]> I saw some related issues in Fedora workstation forums, but the suggested `module_blacklist=ucsi_acpi` kernel option didn't help. I also tried disabling CPU mitigations, adding `rd.shell rd.debug`, nothing seems to work. The same ISO is booting okay on my other laptop.

09:53 <samcday[m]> I thought maybe it was a newer kernel issue, but latest Arch ISO with 6.3.9 is booting fine. Fedora 38 Workstation is also booting fine and showing both early con and full graphical environment.

09:53 <samcday[m]> Any advice/suggestions appreciated!

12:41 <walters> samcday: It may be you're not getting a console, see https://docs.fedoraproject.org/en-US/fedora-coreos/emergency-shell/#_default_console_configuration

13:26 <samcday[m]> Colin Walters: Ah yes, I also tried the suggestions on that page to no avail. Although maybe I've been trying the wrong thing. I haven't been customizing the kernel args per the suggestions on that page, but instead hitting TAB on the GRUB splash and editing the kernel arguments there. I tried to append `console=tty0` using that method with no effect

13:27 <samcday[m]> * that method, with, * no effect. Would you expect that method to work?

13:28 <walters> Yes. You don’t see any logs at all after grub?

13:30 <samcday[m]> Zip. The similar issue reported by folks in Fedora Discourse suggest a single, sad, solitary `_` shows on the screen. In my case the screen goes completely blank. The backlight is still on but absolutely no output.

13:30 <samcday[m]> Right after it finishes the "Loading initrd..." there's a brief ... effect. Kinda like a rapid scroll-up of the kernel + initrd loading messages, but with lots of tearing, and then completely blank.

13:32 <samcday[m]> (But to reiterate, I can get a live Fedora Workstation 38 ISO to boot on this same machine)

15:35 <mbff> Hello all, I'm running into what I think is a race condition on boot between a quadlet container pull and networking/DNS being available. Any tips? I suspect this is an issue with systemd and not podman itself.

15:37 <mbff> For a touch more context: I've tried After/Wants with network-online.target, systemd-resolved.service, nss-lookup.service, etc

15:40 <pemensik[m]> I think podman usually does not include systemd inside container. So would not include systemd-resolved there.

15:42 <pemensik[m]> I am not sure how podman propagates resolv.conf to its containers. Do you start podman containers right after boot? Do they have After=nss-lookup.service?

15:42 <mbff> Take podman out of it (I'm using podman to generate systemd services that run containers) -- how in CoreOS can I guarantee that a service starts after DNS/networking is up?

15:43 <mbff> The exact error I'm getting is "Error: initializing source docker://certbot/dns-route53:v2.6.0: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: no such host"

15:44 <pemensik[m]> After=nsss-lookup.target should be enough on the same machine. Can you check what /etc/resolv.conf contents do they have?

15:45 <mbff> The /etc/resolv.conf looks reasonable

15:46 <mbff> I've tried nss-lookup.target (both as a Wants and Requires)

15:46 <pemensik[m]> does getent ahosts registry-1.docker.io return addresses from it?

15:46 <pemensik[m]> don't use Wants or Requires. They should be started anyway. You need just After=

15:47 <mbff> yes it returns an address (if I login to the machine, but that point the networking/DNS seems up). Tempted to add an exec pre with a sleep for 10s but that feels hacky.

15:48 <mbff> I should also note I use butane/ignition to set up static networking which could be causing issues?

15:48 <mbff> I'm pretty much following this portition of the networking guide to set that up: https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/#_butane_config

15:49 <pemensik[m]> It should be possible as podman run -ti fedora:latest getent hosts fedoraproject.org

15:50 <mbff> Not sure I follow... once the host has networking/DNS I should be good to go to pull a container image

15:52 <mbff> Is there a way to wait until ignition has setup static networking? Is it possible that there is a blip between switching from DHCP (auto) to static on that interface?

15:52 <mbff> Or does that setup happen on the initial boot anyway (before any systemd services I've defined run)

15:52 <pemensik[m]> so it is failing not inside the container, but still on the host preparing that container?

15:52 <pemensik[m]> I don't know anything about ignition, sorry. cannot help with that

15:53 <mbff> correct

15:53 <mbff> the container pull/download is failing

15:53 <pemensik[m]> it should work with NM or systemd-networkd

15:54 <mbff> What do you mean?

15:55 <pemensik[m]> Try comparing error time with time of "systemctl status nss-lookup.target" got started.

15:56 <pemensik[m]> waiting for network-online.target has to be implemented by network configuration system. I have no idea whether ignition tries to do that.

15:56 <mbff> good idea on that first message

15:57 <mbff> As for network-online.. wouldn't that be up to NM to implmement that not ignition?

15:59 <mbff> The log "Reached target nss-lookup.target" happened on the same sec as "Trying to pull docker.io/certbot/dns-route53:v2.6.0..."

15:59 <mbff> Let me try again with nss-lookup.target

16:00 <pemensik[m]> well, maybe. It is generating NM configuration. but not sure it works correctly with method=manual

16:03 <mbff> Still logging on the same second

16:03 <mbff> With After=network-online.target nss-lookup.target Wants=network-online.target nss-lookup.target

16:07 <mbff> Wonder if I shouldn't try After and Wants with NetworkManager-wait-online.service and NetworkManager.service?

16:08 <mbff> Nevermind, NM-wait-online is WantedBy=network-online.target.

16:18 <mbff> Any other ideas?

19:22 <mbff> pemensik[m], still around? I've been troubleshooting some more with restart=on-failure, but that solution doesn't work for oneshot services.

19:23 <mbff> Bottom line: I'm convinced that NetworkManager-wait-online.service starts too quickly that services running after network-online.target may fail if they rely on network/DNS.

19:24 <mbff> and that services*

20:58 <pemensik[m]> Its possible. I expect most software should retry few times before giving up. I admit it might be difficult to analyze it properly. Recording queries by wireshark might help