Good'day
Just out of curiosity I did a tcpdump port 80 on one of my remote virtual machines. I see a huge amount of requests to this http-port, 95% going to some AWS EC2 instance. Strange thing: The Web-Server was not on high load at all. The packets from tcpdump look like this:
20:57:16.860028 IP user123.example.net.http > ec2-55-155-123-123.ap-northeast-2.compute.amazonaws.com.12345: Flags [S.], seq 1234567, ack 12345567, win 12300, options [mss 1460], length 0
(user123.example.net.http is the address of my remote machine)
All of them looks basically the same, only thing that changes, is the actual EC2-adress (aka IP-adress) and the destination port. Sometimes it's an incoming packag (to port 80, too), but most of them are outgoing.
I run this tcpdump on port 80 today, from the office now. Now the traffic is between the remote virtual machine and another IP-adress, not the AWS-instance anymore:
08:34:46.369161 IP user123.example.net.http > 123.12.123.132.37176: Flags [R.], seq 0, ack 2821777444, win 0, length 0
08:34:46.388933 IP 123.12.123.132.55539 > user123.example.net.http : Flags [S], seq 2934790784, win 29200, length 0
In this case, the IP-Adress does not change.
Only thing that changed: I remotely logged in from my working place now, using another IP-Adress (not the one you can see in tcpdump) and another SSH-client.
Investigation
What I tried so far, without success:
- I checked the logfile of nginx, they don't show any traffic (I am not running huges sites, just some little private projects)
- I stopped nginx, the only apparent port80-daemon in this setup
- I stopped all other non-system services: no changes
Current setup
I am running nginx on this server on port 80 and 443, but traffic is generally routed to HTTPS. Also I am running PHP 7.3 with FPM, but on sockets, not via TCP.
Besides that, I am running some e-mail services (dovecot, postfix, ...), Unbound, a *MySQL-*Server and an *ELK-*stack. This is what netstat -tulpn confirms: (I x'ed my public IP-Adress and my SSHD-port, which is not the default one)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 858/unbound
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2806/master
tcp 0 0 x.x.x.x:25 0.0.0.0:* LISTEN 2806/master
tcp 0 0 127.0.0.1:8953 0.0.0.0:* LISTEN 858/unbound
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 7942/nginx -g daemo
tcp 0 0 0.0.0.0:4190 0.0.0.0:* LISTEN 628/dovecot
tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 628/dovecot
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 491/mysqld
tcp 0 0 127.0.0.1:587 0.0.0.0:* LISTEN 2806/master
tcp 0 0 x.x.x.x:587 0.0.0.0:* LISTEN 2806/master
tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 628/dovecot
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 7942/nginx -g daemo
tcp 0 0 0.0.0.0:??? 0.0.0.0:* LISTEN 27952/sshd
tcp6 0 0 ::1:53 :::* LISTEN 858/unbound
tcp6 0 0 ::1:25 :::* LISTEN 2806/master
tcp6 0 0 ::1:8953 :::* LISTEN 858/unbound
tcp6 0 0 :::443 :::* LISTEN 7942/nginx -g daemo
tcp6 0 0 :::4190 :::* LISTEN 628/dovecot
tcp6 0 0 :::993 :::* LISTEN 628/dovecot
tcp6 0 0 ::1:587 :::* LISTEN 2806/master
tcp6 0 0 :::143 :::* LISTEN 628/dovecot
tcp6 0 0 :::80 :::* LISTEN 7942/nginx -g daemo
tcp6 0 0 :::??? :::* LISTEN 27952/sshd
udp 0 0 127.0.0.1:53 0.0.0.0:* 858/unbound
udp6 0 0 ::1:53 :::* 858/unbound
After stopping all of the above mentioned servies, this is the output - Packets on port 80 still remains:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:??? 0.0.0.0:* LISTEN 27952/sshd
tcp6 0 0 :::??? :::* LISTEN 27952/sshd
That's the output of service status-all|grep + (marked some with an * - those didn't stop when stopping the parent process, still trying to figure that out)
[ + ] amavis-mc *
[ + ] amavisd-snmp-subagent *
[ + ] apache-htcacheclean *
[ + ] apparmor
[ + ] cron
[ + ] fail2ban
[ + ] gdomap
[ + ] ip6tables
[ + ] iptables
[ + ] lm-sensors
[ ? ] modules_dep.sh
[ + ] nscd
[ ? ] php-fpm-chroot-setup.sh (make PHP available after reboot)
[ + ] procps
[ + ] quota
[ + ] rc.local
[ + ] resolvconf
[ + ] rsyslog
[ + ] ssh
[ + ] sysstat
[ + ] udev
[ + ] unattended-upgrades
[ + ] urandom
[ + ] uwsgi
Question is..
Why is there traffic on port 80, even if nginx is not running? Where does this traffic comes from (what process)?
If your webserver initiates the outgoing request on port 80, stopping your nginx webserver doesn't necessarily help because many services can initiate an outgoing connection on port 80.
You won't get any more detail with tcpdump because it's on the ethernet level. You need to find the source of the problem. Most of the time, when you see such a high requests of outgoing traffic, it's originating from the outside, executing for example a PHP script on one of your sites. But since you disabled Nginx, this is not obvious. But double check if really nothing is listening on port 80 or 443 anymore when nginx is disabled, since such a PHP file can be called from 443 or any other port your webserver is listening on. Check if something still listens on a specific port with this command:
If really nothing is running on port 80 or 443 anymore and you still see the initiated outgoing port 80, I would block incoming requests, see if it's originating from your webserver itself for sure.
If that's the case, check all running services, because only a running service can then be the initiator. You can also use:
or
Check for suspicious services and try to disable them, see if that helps.