Home / ubuntu / Questions / 1397286
In Process
Simple Fellow
Asked: 2022-03-13 22:50:11 +0800 CST

systemd-udev hogs system cpu at 100%

  • 772

Recently I encountered a problem on one of the nodes of kubernetes where system was running at a crawling speed. after investigating I found that several instances of systemd-udevd were running.

systemctl status systemd-udevd
● systemd-udevd.service - udev Kernel Device Manager
     Loaded: loaded (/lib/systemd/system/systemd-udevd.service; static; vendor preset: enabled)
     Active: active (running) since Sun 2022-03-13 08:32:29 EET; 31s ago
TriggeredBy: ● systemd-udevd-control.socket
             ● systemd-udevd-kernel.socket
       Docs: man:systemd-udevd.service(8)
             man:udev(7)
   Main PID: 541638 (systemd-udevd)
     Status: "Processing with 18 children at max"
      Tasks: 19
     Memory: 18.0M
     CGroup: /system.slice/systemd-udevd.service
             ├─541638 /lib/systemd/systemd-udevd
             ├─541857 /lib/systemd/systemd-udevd
             ├─541858 /lib/systemd/systemd-udevd
             ├─541859 /lib/systemd/systemd-udevd
             ├─541860 /lib/systemd/systemd-udevd
             ├─541862 /lib/systemd/systemd-udevd
             ├─541863 /lib/systemd/systemd-udevd
             ├─541864 /lib/systemd/systemd-udevd
             ├─541865 /lib/systemd/systemd-udevd
             ├─541866 /lib/systemd/systemd-udevd
             ├─541867 /lib/systemd/systemd-udevd
             ├─541868 /lib/systemd/systemd-udevd
             ├─541869 /lib/systemd/systemd-udevd
             ├─541870 /lib/systemd/systemd-udevd
             ├─541871 /lib/systemd/systemd-udevd
             ├─541872 /lib/systemd/systemd-udevd
             ├─541873 /lib/systemd/systemd-udevd
             ├─541874 /lib/systemd/systemd-udevd
             └─541875 /lib/systemd/systemd-udevd

Mar 13 08:32:29 xxxx.cloud systemd[1]: Starting udev Kernel Device Manager...
Mar 13 08:32:29 xxxx.cloud systemd[1]: Started udev Kernel Device Manager.

I tried stopping the service, terminated processes and even disable it but to no avail. it keeps getting spawned and hogs the system completely. I also stopped docker and k3s-agent but the systemd-udevd service just keeps appearing. This is what the journal entries look like:

- Logs begin at Sun 2022-03-13 04:44:26 EET, end at Sun 2022-03-13 08:39:17 EET. --
Mar 13 07:56:39 xxxx.cloud systemd[1]: Stopping udev Kernel Device Manager...
Mar 13 07:56:40 xxxx.cloud systemd[1]: systemd-udevd.service: Succeeded.
Mar 13 07:56:40 xxxx.cloud systemd[1]: Stopped udev Kernel Device Manager.
Mar 13 07:56:40 xxxx.cloud systemd[1]: Starting udev Kernel Device Manager...
Mar 13 07:56:40 xxxx.cloud systemd[1]: Started udev Kernel Device Manager.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Main process exited, code=killed, status=9/KILL
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525892 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525893 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525894 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525896 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525897 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525898 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525899 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525900 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525901 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Killing process 525902 (systemd-udevd) with signal SIGKILL.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Failed with result 'signal'.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 1.
Mar 13 07:57:46 xxxx.cloud systemd[1]: Stopped udev Kernel Device Manager.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Found left-over process 525885 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 13 07:57:46 xxxx.cloud systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Found left-over process 525886 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 13 07:57:46 xxxx.cloud systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Found left-over process 525887 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 13 07:57:46 xxxx.cloud systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Found left-over process 525889 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 13 07:57:46 xxxx.cloud systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 13 07:57:46 xxxx.cloud systemd[1]: systemd-udevd.service: Found left-over process 525890 (systemd-udevd) in control group while starting unit. Ignoring.
Mar 13 07:57:46 xxxx.cloud systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

these are the latest entries from udevadm monitor

DEV  [71139.321376] add      /kernel/slab/:A-0000256/cgroup/filp(183862:containerd.service) (cgroup)
UDEV  [71139.328358] add      /kernel/slab/kmalloc-512/cgroup/kmalloc-512(183862:containerd.service) (cgroup)
UDEV  [71139.335767] add      /kernel/slab/sock_inode_cache/cgroup/sock_inode_cache(183862:containerd.service) (cgroup)
UDEV  [71139.343031] add      /kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(183862:containerd.service) (cgroup)
UDEV  [71139.344456] add      /kernel/slab/:A-0000128/cgroup/pid(183862:containerd.service) (cgroup)
UDEV  [71139.346948] add      /kernel/slab/dentry/cgroup/dentry(183862:containerd.service) (cgroup)
UDEV  [71139.350987] add      /kernel/slab/kmalloc-64/cgroup/kmalloc-64(183862:containerd.service) (cgroup)
UDEV  [71139.354438] add      /kernel/slab/skbuff_head_cache/cgroup/skbuff_head_cache(183862:containerd.service) (cgroup)
UDEV  [71139.355760] add      /kernel/slab/kmalloc-32/cgroup/kmalloc-32(183862:containerd.service) (cgroup)
UDEV  [71139.356428] add      /kernel/slab/:A-0001024/cgroup/PING(183862:containerd.service) (cgroup)
UDEV  [71139.358485] add      /kernel/slab/:A-0000040/cgroup/pde_opener(183862:containerd.service) (cgroup)
UDEV  [71139.362871] add      /kernel/slab/kmalloc-4k/cgroup/kmalloc-4k(183862:containerd.service) (cgroup)
UDEV  [71139.363302] add      /kernel/slab/:A-0000192/cgroup/cred_jar(183862:containerd.service) (cgroup)
UDEV  [71139.397978] add      /kernel/slab/inode_cache/cgroup/inode_cache(183862:containerd.service) (cgroup)
UDEV  [71139.427001] add      /kernel/slab/:A-0000208/cgroup/vm_area_struct(183862:containerd.service) (cgroup)
UDEV  [71139.454891] add      /kernel/slab/:A-0000064/cgroup/anon_vma_chain(183862:containerd.service) (cgroup)
UDEV  [71139.466958] add      /kernel/slab/anon_vma/cgroup/anon_vma(183862:containerd.service) (cgroup)
UDEV  [71139.474103] add      /kernel/slab/mm_struct/cgroup/mm_struct(183862:containerd.service) (cgroup)
UDEV  [71139.741501] remove   /kernel/slab/sock_inode_cache/cgroup/sock_inode_cache(183862:containerd.service) (cgroup)
UDEV  [71139.757452] remove   /kernel/slab/:A-0000192/cgroup/cred_jar(183862:containerd.service) (cgroup)
UDEV  [71139.796371] add      /kernel/slab/kmalloc-rcl-64/cgroup/kmalloc-rcl-64(183862:containerd.service) (cgroup)
UDEV  [71139.804806] remove   /kernel/slab/anon_vma/cgroup/anon_vma(183862:containerd.service) (cgroup)
UDEV  [71139.810391] remove   /kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(183862:containerd.service) (cgroup)
UDEV  [71139.811768] remove   /kernel/slab/kmalloc-64/cgroup/kmalloc-64(183862:containerd.service) (cgroup)
UDEV  [71139.812294] remove   /kernel/slab/:A-0000128/cgroup/pid(183862:containerd.service) (cgroup)
UDEV  [71139.812943] remove   /kernel/slab/inode_cache/cgroup/inode_cache(183862:containerd.service) (cgroup)
UDEV  [71139.813327] remove   /kernel/slab/skbuff_head_cache/cgroup/skbuff_head_cache(183862:containerd.service) (cgroup)
UDEV  [71139.834801] remove   /kernel/slab/kmalloc-4k/cgroup/kmalloc-4k(183862:containerd.service) (cgroup)
UDEV  [71139.835222] add      /kernel/slab/kmalloc-1k/cgroup/kmalloc-1k(183862:containerd.service) (cgroup)
UDEV  [71139.835556] remove   /kernel/slab/dentry/cgroup/dentry(183862:containerd.service) (cgroup)
UDEV  [71139.836431] remove   /kernel/slab/kmalloc-512/cgroup/kmalloc-512(183862:containerd.service) (cgroup)
UDEV  [71139.836938] remove   /kernel/slab/:A-0000256/cgroup/filp(183862:containerd.service) (cgroup)
UDEV  [71139.845860] remove   /kernel/slab/:A-0000040/cgroup/pde_opener(183862:containerd.service) (cgroup)
UDEV  [71139.858633] remove   /kernel/slab/mm_struct/cgroup/mm_struct(183862:containerd.service) (cgroup)
UDEV  [71139.859594] remove   /kernel/slab/:A-0000208/cgroup/vm_area_struct(183862:containerd.service) (cgroup)
UDEV  [71139.860200] remove   /kernel/slab/:A-0000064/cgroup/anon_vma_chain(183862:containerd.service) (cgroup)
UDEV  [71139.921380] remove   /kernel/slab/kmalloc-32/cgroup/kmalloc-32(183862:containerd.service) (cgroup)
UDEV  [71139.925817] remove   /PING(183862:containerd.service) (cgroup)
UDEV  [71139.926651] remove   /kernel/slab/kmalloc-1k/cgroup/kmalloc-1k(183862:containerd.service) (cgroup)
UDEV  [71139.929972] remove   /kernel/slab/kmalloc-rcl-64/cgroup/kmalloc-rcl-64(183862:containerd.service) (cgroup)

Can anyone help me understand whats going on and how can I fix this?

docker

0 Answers