I noticed one of the servers I manage has been showing traffic on nethogs from external IPs - several yesterday from Italy, and several today from Korea. I shut down all services on the server that should be creating external traffic and they still showed up. However upon further examination, I can't find connections in netstat.
The below is edited to show the server IP 1.1.1.1 on the 1.1.1.0/24 network.
nethogs looks something like this:
PID USER PROGRAM DEV SENT RECEIVED
12345 root /user/bin/php eth0 7266 25921 B
12346 me sshd: me@pts/1 0 22200 B
? root 1.1.1.1:443-112.175.124.2:42910 0.0 4320 B
And a few others to devices on the LAN that I expect to be present...
But when I run netstat -antpeu it looks like this:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 780385 28327/cupsd
tcp 0 0 0.0.0.0:1234 0.0.0.0:* LISTEN 0 780488 28385/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 13072 2492/master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 0 732393 23099/openvpn
tcp 0 0 0.0.0.0:3551 0.0.0.0:* LISTEN 0 12007 2256/apcupsd
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 27 12857 2397/mysqld
tcp 0 0 0.0.0.0:58380 0.0.0.0:* LISTEN 29 10990 1973/rpc.statd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 10741 1913/rpcbind
tcp 0 0 1.1.1.1:1234 1.1.1.112:50248 ESTABLISHED 0 749495 24300/sshd
tcp 0 0 ::1:631 :::* LISTEN 0 780384 28327/cupsd
tcp 0 0 :::1234 :::* LISTEN 0 780490 28385/sshd
tcp 0 0 ::1:25 :::* LISTEN 0 13073 2492/master
tcp 0 0 :::444 :::* LISTEN 0 13725 2605/httpd
tcp 0 0 :::47266 :::* LISTEN 29 10996 1973/rpc.statd
tcp 0 0 :::111 :::* LISTEN 0 10744 1913/rpcbind
tcp 0 0 :::80 :::* LISTEN 0 13721 2605/httpd
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.1:34570 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.31:55578 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.31:55584 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.35:58086 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.35:58084 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.35:58088 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.35:58080 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.31:55576 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.31:55582 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.1:34568 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.31:55580 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.31:55586 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.35:58082 TIME_WAIT 0 0 -
tcp 0 0 ::ffff:1.1.1.1:80 ::ffff:1.1.1.35:58090 TIME_WAIT 0 0 -
udp 0 0 0.0.0.0:44358 0.0.0.0:* 29 10987 1973/rpc.statd
udp 0 0 127.0.0.1:877 0.0.0.0:* 0 10982 1973/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:* 0 10739 1913/rpcbind
udp 0 0 0.0.0.0:631 0.0.0.0:* 0 780388 28327/cupsd
udp 0 0 10.8.0.1:123 0.0.0.0:* 38 732403 2243/ntpd
udp 0 0 1.1.1.1:123 0.0.0.0:* 0 11996 2243/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 0 11995 2243/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 0 11988 2243/ntpd
udp 0 0 0.0.0.0:816 0.0.0.0:* 0 10740 1913/rpcbind
udp 0 0 :::34521 :::* 29 10993 1973/rpc.statd
udp 0 0 :::111 :::* 0 10742 1913/rpcbind
udp 0 0 fe80::21a:a0ff:fe56:5d2:123 :::* 0 11999 2243/ntpd
udp 0 0 ::1:123 :::* 0 11998 2243/ntpd
udp 0 0 :::123 :::* 0 11989 2243/ntpd
udp 0 0 :::816 :::* 0 10743 1913/rpcbind
So what is going on here? 1) Has the server been compromised? It looks like something is generating traffic to the outside that shouldn't be... I'm also suspicious because when I block one of the IPs via the firewall and the router a new one comes up.
2) Why are the connections to the external IPs not showing in netstat?
3) Is there any way to track down which PID they are coming from? nethogs does not show it and neither does iftop.
EDIT: I should mention the 2 ports it is showing up on are 443 and 4040. Even when I try closing these in iptables the connections show up.
One other way of locating the PID is using the 'lsof' command ie: lsof -i :443
Yet another way is using the 'ss' command ie: ss -o '( sport = :443 )' using 'dport' for destination port and 'sport' for source
or the 'fuser' command ie: 'fuser tcp/443'
Have you tried looking into the IPs? Identifying any possible services on the other side that this connection could be going to? Are the SRC/DST ports ever the same?
You can fire up tcpdump or wireshark and start collecting a packet capture, then block the already established connections to force a new one to open as you did before.
I would expect the traffic to be SSL (443), but you won't truly know until you examine it. If it is SSL and you really want to see what it is, you can try performing a MITM capture.