Ping a Specific Port

Question

RobbieTheK

Asked: 2020-06-30 20:06:24 +0800 CST2020-06-30 20:06:24 +0800 CST 2020-06-30 20:06:24 +0800 CST

tracking down a bash process for a df command

772

I’m using the tips on this SO thread.

ps -awux|grep df
root       15826  0.0  0.0      0     0 ?        I<   May22   0:00 [cifs-dfscache]
myuser 3086246  0.0  0.0 216860  3212 ?        Ss   16:06   0:02 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df;echo '==> /proc/who <==';who;echo '==> /proc/end <==';echo '##Moba##'; done
myuser 3137650  0.0  0.0 215348   616 ?        D    16:27   0:00 df

So I cat the process 3137650 stack:

cat /proc/3137650/stack
[<0>] autofs_wait+0x25b/0x723
[<0>] autofs_mount_wait+0x49/0xf0
[<0>] autofs_d_automount+0xdb/0x200
[<0>] follow_managed+0x110/0x2c0
[<0>] walk_component+0x1e9/0x2f0
[<0>] path_lookupat+0x70/0x120
[<0>] filename_lookup+0x97/0x180
[<0>] user_statfs+0x33/0xa0
[<0>] __do_sys_statfs+0x10/0x30
[<0>] do_syscall_64+0x5b/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

And the other process id 3086246:

cat /proc/3086246/stack
[<0>] do_wait+0x1b3/0x220
[<0>] kernel_wait4+0x96/0x120
[<0>] do_syscall_64+0x5b/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Then:

xargs -0 -n 1 echo < /proc/3137650/environ
SHELL=/bin/bash
MATHEMATICA_HOME=/usr/local/Wolfram/Mathematica/11.3
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
XDG_CONFIG_HOME=/home/myuser/.config
SPARK_LOCAL_HOSTNAME=localhost
LMOD_DIR=/usr/share/lmod/lmod/libexec
PWD=/home/myuser
LOGNAME=myuser
XDG_SESSION_TYPE=tty
MODULESHOME=/usr/share/lmod/lmod
MANPATH=/usr/share/lmod/lmod/share/man:
CUDA_INCLUDE_DIRS=/usr/include/cuda
SPARK_MASTER_IP=127.0.0.1
HOME=/home/myuser
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
LANG=en_US.UTF-8
XDG_CONFIG_DIR=/home/myuser/.config
LMOD_SETTARG_FULL_SUPPORT=no
CUDA_INC_PATH=/usr/include/cuda
LMOD_VERSION=8.2.10
SSH_CONNECTION=x.x.x.x 51696 x.x.x.x 22
MODULEPATH_ROOT=/usr/share/modulefiles
XDG_SESSION_CLASS=user
LMOD_PKG=/usr/share/lmod/lmod
HADOOP_HOME=/usr/local/bin/hadoop-2.9.0
GUROBI_HOME=/home/student/gurobi811/linux64/
LESSOPEN=||/usr/bin/lesspipe.sh %s
USER=kudyba
LMOD_ROOT=/usr/share/lmod
SHLVL=1
BASH_ENV=/usr/share/lmod/lmod/init/bash
LMOD_sys=Linux
SPARK_HOME=/usr/local/bin/spark
SPARK_LOCAL_IP=127.0.0.1
XDG_SESSION_ID=7778
LD_LIBRARY_PATH=:/home/student/gurobi811/linux64//lib
XDG_RUNTIME_DIR=/run/user/6105
SSH_CLIENT=x.x.x.x 51696 22
PIG_INSTALL=/usr/local/bin/pig-0.17.0
SPARK_EXAMPLES_JAR=/usr/local/bin/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jar
KDEDIRS=/usr
XDG_DATA_DIRS=/home/myuser/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
PATH=/usr/local/bin/anaconda3/bin:/home/users/mzilversmit/ncbi-blast-2.7.1+/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/users/students/mchen177/gurobi811/linux64//bin:/usr/local/bin/spark/bin:/usr/local/bin/pig-0.17.0/bin:/opt/dell/srvadmin/bin:/usr/local/bin/spark/bin
MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
SPARK_CLASSPATH=/usr/share/java/mysql-connector-java.jar
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/6105/bus
LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod
BASH_FUNC_ml%%=() {  eval $($LMOD_DIR/ml_cmd "$@")
}
BASH_FUNC_module%%=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
_=/usr/bin/df

and from the other process:

xargs -0 -n 1 echo < /proc/3086246/environ
USER=myuser
LOGNAME=myuser
HOME=/home/myuser
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
SHELL=/bin/bash
XDG_SESSION_ID=7778
XDG_RUNTIME_DIR=/run/user/xxxx
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/xxxx/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
SSH_CLIENT=edited 51696 22
SSH_CONNECTION=edited 51696 edited 22

And just to confirm the parent process id:

ps --ppid  3086246
    PID TTY          TIME CMD
3137650 ?        00:00:00 df

Not sure if the echo ##Moba## is a clue, as I’m using Mobaxterm.

When I try a strace of the parent process it just waits at:

strace -p  3086246
strace: Process 3086246 attached
wait4(-1,

However the child process returns:

strace -p 3137650
strace: attach: ptrace(PTRACE_SEIZE, 3137650): Operation not permitted

For good measure:

pstree -pl 3137650
df(3137650)

and:

pstree -pl 3086246
bash(3086246)───df(3137650)

and:

ps -fp 3086246
UID          PID    PPID  C STIME TTY          TIME CMD
myuser   3086246       1  0 16:06 ?        00:00:02 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head

So TTY is a '?', nothing from cronjobs that I could see.

Do the 2 different status of Ss for 3086246 and D for 3137650, nice table here:

D    uninterruptible sleep (usually IO)
S    interruptible sleep (waiting for an event to complete)
s    is a session leader

I also tried a gdb on the 3086246 PID:

Reading symbols from /usr/bin/bash...
Reading symbols from /usr/lib/debug/usr/bin/bash-5.0.17-1.fc32.x86_64.debug...
Reading symbols from /lib64/libtinfo.so.6...
Reading symbols from /usr/lib/debug/usr/lib64/libtinfo.so.6.1-6.1-15.20191109.fc32.x86_64.debug...
Reading symbols from /lib64/libdl.so.2...
Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.31.so.debug...
Reading symbols from /lib64/libc.so.6...
Reading symbols from /usr/lib/debug/usr/lib64/libc-2.31.so.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/usr/lib64/ld-2.31.so.debug...
0x00007f8e059ccf3a in __GI___wait4 (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7fffa494ff10, options=options@entry=0,
    usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
27        return SYSCALL_CANCEL (wait4, pid, stat_loc, options, usage);

Any thoughts or other debugging commands to try?

Edit: added systemctl status of PIDs per @michael-hampton suggestion:

systemctl status 3086246 -l --no-pager
● session-7778.scope - Session 7778 of user myuser
     Loaded: loaded (/run/systemd/transient/session-7778.scope; transient)
  Transient: yes
     Active: active (abandoned) since Mon 2020-06-29 13:03:32 EDT; 11h ago
      Tasks: 2
     Memory: 316.9M
        CPU: 11min 46.751s
     CGroup: /user.slice/user-6105.slice/session-7778.scope
             ├─3086246 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df;echo '==> /proc/who <==';who;echo '==> /proc/end <==';echo '##Moba##'; done
             └─3137650 df

Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           20 May 29 14:35 usr/share/unimaps -> /usr/lib/kbd/unimaps
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   3 root     root            0 May 29 14:35 var
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           11 May 29 14:35 var/lock -> ../run/lock
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root            6 May 29 14:35 var/run -> ../run
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   2 root     root            0 May 29 14:35 var/tmp
Jun 29 16:26:21 ourserver  dracut[3126800]: ========================================================================
Jun 29 16:26:21 ourserver  dracut[3126800]: *** Creating initramfs image file '/boot/initramfs-5.6.19-300.fc32.x86_64.tmp' done ***
Jun 29 16:27:21 ourserver  systemd-tmpfiles[3137785]: /usr/lib/tmpfiles.d/lxdm.conf:1: Line references path below legacy directory /var/run/, updating /var/run/lxdm → /run/lxdm; please update the tmpfiles.d/ drop-in file accordingly.
Jun 29 16:48:05 ourserver  su[2983955]: pam_unix(su:session): session closed for user root
Jun 29 16:48:07 ourserver  sshd[2983731]: pam_unix(sshd:session): session closed for user myuser

and:

systemctl status 3137650 -l --no-pager
● session-7778.scope - Session 7778 of user myuser
     Loaded: loaded (/run/systemd/transient/session-7778.scope; transient)
  Transient: yes
     Active: active (abandoned) since Mon 2020-06-29 13:03:32 EDT; 11h ago
      Tasks: 2
     Memory: 316.9M
        CPU: 11min 46.751s
     CGroup: /user.slice/user-6105.slice/session-7778.scope
             ├─3086246 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df;echo '==> /proc/who <==';who;echo '==> /proc/end <==';echo '##Moba##'; done
             └─3137650 df

Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           20 May 29 14:35 usr/share/unimaps -> /usr/lib/kbd/unimaps
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   3 root     root            0 May 29 14:35 var
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           11 May 29 14:35 var/lock -> ../run/lock
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root            6 May 29 14:35 var/run -> ../run
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   2 root     root            0 May 29 14:35 var/tmp
Jun 29 16:26:21 ourserver  dracut[3126800]: ========================================================================
Jun 29 16:26:21 ourserver  dracut[3126800]: *** Creating initramfs image file '/boot/initramfs-5.6.19-300.fc32.x86_64.tmp' done ***
Jun 29 16:27:21 ourserver  systemd-tmpfiles[3137785]: /usr/lib/tmpfiles.d/lxdm.conf:1: Line references path below legacy directory /var/run/, updating /var/run/lxdm → /run/lxdm; please update the tmpfiles.d/ drop-in file accordingly.
Jun 29 16:48:05 ourserver  su[2983955]: pam_unix(su:session): session closed for user root
Jun 29 16:48:07 ourserver  sshd[2983731]: pam_unix(sshd:session): session closed for user myuser

I'm starting to think it might be this bug except that ssh is not slow. Running:

systemctl | grep "abandoned" | grep -e "-[[:digit:]]"

returned several abandoned ssh sessions. Running:

systemctl | grep "abandoned" | grep -e "-[[:digit:]]" | sed "s/.scope.*/.scope/" | xargs systemctl stop

removes all of the abandoned sessions and the 'df' command disappears from 'ps'.

1 Answers

Voted

RobbieTheK · Answer 1 · 2020-07-02T06:19:56+08:00

RobbieTheK

2020-07-02T06:19:56+08:002020-07-02T06:19:56+08:00

Mystery solved. It is indeed MobaXterm, bottom left, there is a Remote Monitoring option:

1

tracking down a bash process for a df command

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?