systemd-networkd-wait-online service timing out during boot 22.04.4 LTS

[JayCroghan]

I have this issue with systemd-networkd-wait-online.service timing out during boot but the fix hasn't been a single one of the myriad of fixes I have found online.

Let me start at the start:

I have been this issue for quite a while so today I have tried a suggestion I found online and running systemd-analyze plot and checking the output.


I'm attaching the SVG file here because unless I am mistaken there are only two things taking an absurdly long time to boot, one of them systemd-networkd-wait-online (of course) but then afterwards, strangely, sendmail which I do not even use on this server it is within my home network and only functions as a seedbox/media server.


Am I missing something here? I disabled wlan previously to rule it out as it's a wired machine and has no need for wlan.


https://c0ld.net/slowboot.svg


I disabled sendmail, rebooted, watched it timeout and then ran it again and now the only thing taking time is the thing I'm trying to fix.


https://c0ld.net/slowboot-nosendmail.svg


Here is a full journalctl: http://0x0.st/Xo36.txt


It has a static IP, it is not internet facing, runs plex, bittorrent-nox, jellyfin and has a couple of USB hard drives mounted at boot for storage and that's about all that's running other than webmin and cockpit. It has the full Ubuntu GNome installed but not booting at startup, I just wanted it for kicks and giggles in case if I need to ever use it as a backup PC.


I really do not want to simply mask the problem by disabling the service. I even tried changing to renderer: networkd from renderer: NetworkManager in netplan


Output of networkctl, wlan is not configured on purpose.


IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eno1 ether routable configured
3 wlp1s0 wlan off unmanaged


3 links listed.


So nothing crazy there.


I also have optional: true under config of eno1 in netplan, changing it to false makes no difference.


I also tried adding --any and then --interface=eno1 as specified below (restarting the service both times) and both still have the same timeout.
https://ubuntuforums.org/showthread.php?t=2490962

Apr 21 15:15:16 jays-lenovo systemd[1]: Starting Wait for Network to be Configured...
░░ Subject: A start job for unit systemd-networkd-wait-online.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit systemd-networkd-wait-online.service has begun execution.
░░
░░ The job identifier is 3886.
Apr 21 15:17:16 jays-lenovo systemd-networkd-wait-online[4967]: Timeout occurred while waiting for network connectivity.
Apr 21 15:17:16 jays-lenovo systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit systemd-networkd-wait-online.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 21 15:17:16 jays-lenovo systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit systemd-networkd-wait-online.service has entered the 'failed' state with result 'exit-code'.
Apr 21 15:17:16 jays-lenovo systemd[1]: Failed to start Wait for Network to be Configured.
░░ Subject: A start job for unit systemd-networkd-wait-online.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit systemd-networkd-wait-online.service has finished with a failure.
░░
░░ The job identifier is 3886 and the job result is failed.

This was after restarting the service with --interface=eno1 which is most definitely up!!!!


The same check for systemd-networkd.service

-- Boot 984031206640442cb02a40798ddf3b7f --
Apr 21 14:21:12 jays-lenovo systemd[1]: Starting Network Configuration...
░░ Subject: A start job for unit systemd-networkd.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit systemd-networkd.service has begun execution.
░░
░░ The job identifier is 49.
Apr 21 14:21:12 jays-lenovo systemd-networkd[1178]: lo: Link UP
Apr 21 14:21:12 jays-lenovo systemd-networkd[1178]: lo: Gained carrier
Apr 21 14:21:12 jays-lenovo systemd-networkd[1178]: Enumeration completed
Apr 21 14:21:12 jays-lenovo systemd[1]: Started Network Configuration.
░░ Subject: A start job for unit systemd-networkd.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit systemd-networkd.service has finished successfully.
░░
░░ The job identifier is 49.
Apr 21 14:23:15 jays-lenovo systemd-networkd[1178]: eno1: Link UP
Apr 21 14:23:18 jays-lenovo systemd-networkd[1178]: eno1: Gained carrier
Apr 21 14:23:19 jays-lenovo systemd-networkd[1178]: eno1: Gained IPv6LL
Apr 21 14:51:50 jays-lenovo systemd-networkd[1178]: eno1: Re-configuring with /run/systemd/network/10-netplan-eno1.network
Apr 21 14:51:51 jays-lenovo systemd-networkd[1178]: eno1: Re-configuring with /run/systemd/network/10-netplan-eno1.network
Apr 21 14:51:51 jays-lenovo systemd-networkd[1178]: eno1: DHCPv6 lease lost
Apr 21 14:52:23 jays-lenovo systemd[1]: Stopping Network Configuration...
░░ Subject: A stop job for unit systemd-networkd.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit systemd-networkd.service has begun execution.
░░
░░ The job identifier is 4966.
Apr 21 14:52:23 jays-lenovo systemd-networkd[1178]: eno1: DHCPv6 lease lost
Apr 21 14:52:23 jays-lenovo systemd[1]: systemd-networkd.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit systemd-networkd.service has successfully entered the 'dead' state.
Apr 21 14:52:23 jays-lenovo systemd[1]: Stopped Network Configuration.
░░ Subject: A stop job for unit systemd-networkd.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit systemd-networkd.service has finished.
░░
░░ The job identifier is 4966 and the job result is done.
-- Boot dc2cb94cb1f649c49648ba8e085192db --
Apr 21 14:53:19 jays-lenovo systemd[1]: Starting Network Configuration...
░░ Subject: A start job for unit systemd-networkd.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit systemd-networkd.service has begun execution.
░░
░░ The job identifier is 25.
Apr 21 14:53:19 jays-lenovo systemd-networkd[1172]: lo: Link UP
Apr 21 14:53:19 jays-lenovo systemd-networkd[1172]: lo: Gained carrier
Apr 21 14:53:19 jays-lenovo systemd-networkd[1172]: Enumeration completed
Apr 21 14:53:19 jays-lenovo systemd[1]: Started Network Configuration.
░░ Subject: A start job for unit systemd-networkd.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit systemd-networkd.service has finished successfully.
░░
░░ The job identifier is 25.
Apr 21 14:53:19 jays-lenovo systemd-networkd[1172]: eno1: Link UP
Apr 21 14:53:22 jays-lenovo systemd-networkd[1172]: eno1: Gained carrier
Apr 21 14:53:24 jays-lenovo systemd-networkd[1172]: eno1: Gained IPv6LL
Apr 21 15:26:32 jays-lenovo systemd-networkd[1172]: eno1: Re-configuring with /run/systemd/network/10-netplan-eno1.network
Apr 21 15:26:32 jays-lenovo systemd-networkd[1172]: eno1: DHCPv6 lease lost
Apr 21 15:26:32 jays-lenovo systemd-networkd[1172]: eno1: Re-configuring with /run/systemd/network/10-netplan-eno1.network
Apr 21 15:26:32 jays-lenovo systemd-networkd[1172]: eno1: DHCPv6 lease lost

Full https://c0ld.net/dmesg.html available here. I do not know what else to try at this point, I have scoured the internet.

eno1 is configured as a static IP too so it makes no sense it's taking longer to be available than shorter.
192.168.2.1 is the main router for my entire household with multiple different routers and tens of devices, not a single other one has any issue.

# This is the network config written by 'subiquity'
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      optional: false
      addresses:
        - 192.168.2.99/24
      nameservers:
        addresses: [192.168.2.1]
      routes:
        - to: default
          via: 192.168.2.1

[VMC]

Read this link on systemd-networkd-wait-online.service and apply suggested fix.

[JayCroghan]

Are you referring specifically to removing the couple of seconds by disabling those ccouple of unnecesary services? Considering the timeout is in minutes and this server isn't really doing much, even though I knew this was going to do nothing I went ahead anyway. Similar to the person in that thread, this is after removing those services:

systemd-analyze critical-chain

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.


multi-user.target @2min 19.505s
└─snapd.seeded.service @2min 6.485s +12.979s
  └─basic.target @2min 6.389s
    └─sockets.target @2min 6.384s
      └─cockpit.socket @2min 6.324s +56ms
        └─sysinit.target @2min 6.213s
          └─cloud-init.service @2min 5.715s +491ms
            └─cloud-init-local.service @4.148s +1.183s
              └─systemd-remount-fs.service @1.200s +106ms
                └─systemd-journald.socket @1.015s
                  └─system.slice @967ms
                    └─-.slice @967ms

All of them are ion seconds, but beging @2 min+. The services are not the issue. My home network isn't the issue. Something is causing my eth0 (which for some reason is renamed eno1 and not by me) to take 2+ mins to get a link on a hard wired 1 Gb wired connection. I can internet test around 850Mbs so it's not just link speed.

I read the whole rest of that thread in case I am missing something silly, and I'm not sure I am? I summed up the entire list of systemd-analyze blame which is 106 seconds plus 2000ms~ so 108 seconds. That is still leaving something like 1 minute 40 seconds of time unaccounted for?

Here is the full blame anyway:

2min 100ms systemd-networkd-wait-online.service
   11.436s snapd.seeded.service
   11.294s snapd.service
   10.711s snap.lxd.activate.service
    5.624s webmin.service
    4.886s postgresql@14-main.service
    3.553s containerd.service
    2.569s dev-mapper-ubuntu\x2d\x2dvg\x2dubuntu\x2d\x2dlv.device
    2.165s nmbd.service
    2.000s pmlogger.service
    1.889s dev-loop14.device
    1.888s dev-loop12.device
    1.886s dev-loop8.device
    1.871s qbittorrent-nox.service
    1.863s dev-loop11.device
    1.851s dev-loop17.device
    1.851s dev-loop15.device
    1.827s dev-loop13.device
    1.827s dev-loop16.device
    1.824s dev-loop18.device
    1.792s pmcd.service
    1.790s dev-loop9.device
    1.786s dev-loop19.device
    1.651s dev-loop10.device
    1.597s dev-loop20.device
    1.508s xrdp.service
    1.389s dev-loop7.device
    1.389s dev-loop5.device
    1.370s dev-loop6.device
    1.364s dev-loop4.device
    1.363s dev-loop3.device
    1.310s dev-loop2.device
    1.292s pmie.service
    1.264s cloud-init-local.service
    1.235s dev-loop1.device
    1.234s dev-loop0.device
    1.084s cups.service
     996ms networkd-dispatcher.service
     928ms ufw.service
     885ms systemd-journal-flush.service
     720ms winbind.service
     716ms apparmor.service
     704ms snapd.apparmor.service
     662ms nfs-server.service
     584ms bluetooth.service
     577ms avahi-daemon.service
     518ms smbd.service
     512ms cloud-init.service
     486ms thermald.service
     474ms wpa_supplicant.service
     450ms media-ssd.mount
     447ms systemd-logind.service
     444ms cloud-final.service
     432ms cloud-config.service
     365ms systemd-udevd.service
     364ms cockpit-motd.service
     356ms ssh.service
     352ms apport.service
     325ms media-ssd1.mount
     299ms systemd-rfkill.service
     297ms media-usb.mount
     293ms multipathd.service
     290ms plexmediaserver.service
     281ms snap-bare-5.mount
     275ms snap-core18-2796.mount
     275ms kerneloops.service
     273ms xrdp-sesman.service
     271ms snap-core18-2812.mount
     270ms rpcbind.service
     264ms snap-core20-2182.mount
     258ms snap-core20-2264.mount
     256ms user@1000.service
     253ms snap-core22-1033.mount
     248ms snap-core22-1122.mount
     248ms nfs-mountd.service
     244ms grub-common.service
     243ms systemd-tmpfiles-setup.service
     242ms noip2.service
     236ms snap-cups-1041.mount
     225ms snap-cups-1044.mount
     220ms systemd-udev-trigger.service
     217ms snap-firefox-4090.mount
     215ms atd.service
     211ms snap-firefox-4136.mount
     205ms snap-gnome\x2d3\x2d38\x2d2004-140.mount
     202ms pmproxy.service
     201ms snap-gnome\x2d3\x2d38\x2d2004-143.mount
     199ms systemd-timesyncd.service
     194ms systemd-oomd.service
     190ms snap-gnome\x2d42\x2d2204-172.mount
     179ms snap-gnome\x2d42\x2d2204-176.mount
     177ms openvpn.service
     172ms systemd-resolved.service
     169ms snap-gtk\x2dcommon\x2dthemes-1535.mount
     165ms rsyslog.service
     164ms lvm2-pvscan@8:51.service
     158ms snap-lxd-27428.mount
     153ms nfs-idmapd.service
     152ms alsa-restore.service
     150ms modprobe@drm.service
     148ms snap-lxd-27948.mount
     147ms modprobe@configfs.service
     146ms keyboard-setup.service
     143ms modprobe@fuse.service
     138ms lvm2-monitor.service
     137ms snap-snapd-21184.mount
     134ms systemd-fsck@dev-disk-by\x2duuid-ee2ad6ce\x2da9cc\x2d43ec\x2dbf44\x2d50d0f5afba29.service
     133ms sys-kernel-tracing.mount
     133ms kmod-static-nodes.service
     130ms systemd-binfmt.service
     129ms sys-kernel-debug.mount
     129ms snap-snapd-21465.mount
     129ms rpc-statd.service
     128ms systemd-sysusers.service
     125ms proc-fs-nfsd.mount
     123ms dev-mqueue.mount
     123ms snap-ssd\x2dbenchmark-118.mount
     116ms systemd-modules-load.service
     115ms systemd-sysctl.service
     112ms nfsdcld.service
     112ms systemd-remount-fs.service
     109ms dev-hugepages.mount
     107ms var-snap-firefox-common-host\x2dhunspell.mount
     103ms plymouth-read-write.service
     102ms systemd-tmpfiles-setup-dev.service
      96ms packagekit.service
      92ms systemd-networkd.service
      91ms systemd-journald.service
      81ms grub-initrd-fallback.service
      68ms systemd-user-sessions.service
      67ms systemd-update-utmp.service
      64ms console-setup.service
      64ms sys-fs-fuse-connections.mount
      63ms swap.img.swap
      63ms finalrd.service
      61ms proc-sys-fs-binfmt_misc.mount
      60ms sys-kernel-config.mount
      54ms cockpit.socket
      41ms systemd-random-seed.service
      37ms setvtrgb.service
      36ms rpc-statd-notify.service
      33ms run-rpc_pipefs.mount
      18ms nfs-blkmap.service
      18ms pmlogger_check.service
      17ms plymouth-quit.service
      17ms pmie_check.service
      15ms user-runtime-dir@1000.service
      13ms rtkit-daemon.service
      12ms polkit.service
      11ms boot.mount
      11ms pmie_farm.service
       9ms snapd.socket
       8ms pmlogger_farm.service
       8ms modprobe@efi_pstore.service
       7ms systemd-update-utmp-runlevel.service
       4ms pmlogger_farm_check.service
       4ms pmie_farm_check.service
       1ms postgresql.service

I changed optional to yes instead of true in netplan and nothing changed, I have disabled wifi already and there no other interfaces. There is something fundamentally wrong which is making

systemd-networkd-wait-online.service

take over 2 minutes.

Just to re-iterate, if the system was taking 3 minutes to boot because of things it needed to do then so be it, it's not, it's taking almost 3 minutes to wait for the network interface and from everything in my first post and my config there is no reason it should be.

[JayCroghan]

I have also tried everything here except disabling the service itself: https://askubuntu.com/questions/9722...tu-server-17-1

[1fallen]

Disabling the service will do nothing to help here. Ive tried it previous versions and Usually an update fix's it

Details of the Request:
URL: man:NetworkManager-wait-online.service(8)
Protocol: man
Date and Time: Sunday, April 21, 2024 11:03:51 AM MDT
Additional Information: Unknown protocol 'man'.
Description:
The KIO worker which provides access to the man protocol could not be started. This is usually due to technical reasons.
Possible Causes:
klauncher could not find or start the plugin which provides the protocol. This means you may have an outdated version of the plugin.
Possible Solutions:
Update your software to the latest version. Your distribution should provide tools to update your software.
Contact your appropriate computer support system, whether the system administrator, or technical support group for further assistance.

I'm sorry to hear you are affected by this annoyance though.

[JayCroghan]

How did you get the details of the problem? I can't figure out anything other than "something is making networkd-wait-online.service" from the literal ocean of things I've read today.

[JayCroghan]

Oh wow, ok, took some jigging, for future travellers, it was a mess of NetworkManager and netplan and networkd all conjoined. I went and followed (or did the bits missing anyway) of these two and it all works perfectly, few seconds reboot

https://linux.fernandocejas.com/docs...stemd-networkd

https://askubuntu.com/questions/1336247/removing-netplan-to-use-systemd-networkd-directly

[1fallen]

Oh wow, ok, took some jigging, for future travellers, it was a mess of NetworkManager and netplan and networkd all conjoined. I went and followed (or did the bits missing anyway) of these two and it all works perfectly, few seconds reboot

https://linux.fernandocejas.com/docs...stemd-networkd

https://askubuntu.com/questions/1336247/removing-netplan-to-use-systemd-networkd-directly

Nice I'm saving this if it reappears in the future.

Sorry late reply, Ive been outside in some badly needed Vitamin D (SunShine)

How did you get the details of the problem? I can't figure out anything other than "something is making networkd-wait-online.service" from the literal ocean of things I've read today.

Like this

systemctl status  NetworkManager-wait-online.service
○ NetworkManager-wait-online.service - Network Manager Wait Online
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager-wait-online.service>
     Active: inactive (dead)
       Docs: man:NetworkManager-wait-online.service(8)

[JayCroghan]

Sorry, totally wrong, someone told me change the timeout to 15 seconds and it's just hitting that instead, still timing out at 3 minutes when I remove the 15 second one. So still have this issue.

[JayCroghan]

Sorry, totally wrong, someone told me change the timeout to 15 seconds and it's just hitting that instead, still timing out at 3 minutes when I remove the 15 second one. So still have this issue.