Xorg high CPU usage when session is unlocked + USB port

I'm experiencing Xorg high CPU usage(90-100%) problem when I unlock session. It keeps going for 5 minutes.

Process that causes problem:

/usr/lib/xorg/Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch

Everything seems working normal except Application Shortcuts and it's very annoying. If I press shortcut to launch terminal it will launch with very long delay(~2-3 minutes).

System:

OS: Xubuntu 18.04
Nvidia Driver Version: 390.77
lightdm 1.26.0

EDIT:

I just found out that same behaviour happens when I plug in USB hub or USB IDE/SATA adapter. 100% CPU keeps for 2-3 minutes. Though it's not happening when I insert USB flash drive.