I am having a Ubuntu 10.04 Server with Shorewall 4.4.6.
For some days, i have been seeing that the logs shorewall.log, kernel.log and syslog getting to huge sizes above 20G per log. It occupied all the free disk space, Server become dread slow and all the sites hosted on it stopped working. I had to put a auto-empty of logs every 30 mins.
While exploring these logs i found lot of packets being dropped, from a unknown program sending to a specific IP y.y.y.y, originating from the Ubuntu 10.04 server x.x.x.x (IPs masked for security reasons).
Here is some of the log entries -
Oct 25 12:17:35 web-server kernel: [18401369.775248] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7851 PROTO=UDP SPT=46899 DPT=14000 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775356] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7852 PROTO=UDP SPT=47578 DPT=16413 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775464] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7853 PROTO=UDP SPT=60750 DPT=14557 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775572] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7854 PROTO=UDP SPT=56465 DPT=22698 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775680] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7855 PROTO=UDP SPT=39699 DPT=56776 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775790] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7856 PROTO=UDP SPT=40388 DPT=49843 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775897] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7857 PROTO=UDP SPT=40385 DPT=47112 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776004] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7858 PROTO=UDP SPT=52745 DPT=21869 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776112] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7859 PROTO=UDP SPT=48034 DPT=33058 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776220] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7860 PROTO=UDP SPT=60825 DPT=33964 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776331] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7861 PROTO=UDP SPT=56701 DPT=17518 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776442] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7862 PROTO=UDP SPT=49237 DPT=21521 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776551] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7863 PROTO=UDP SPT=47788 DPT=37887 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776660] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7864 PROTO=UDP SPT=52436 DPT=12071 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776770] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7865 PROTO=UDP SPT=52870 DPT=27053 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776880] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7866 PROTO=UDP SPT=59962 DPT=14336 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776992] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7867 PROTO=UDP SPT=56726 DPT=39180 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.777137] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7868 PROTO=UDP SPT=40108 DPT=14175 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.777249] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7869 PROTO=UDP SPT=55149 DPT=1090 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.777357] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7870 PROTO=UDP SPT=52778 DPT=58315 LEN=8200
I was able to locate the program using 'netstat', something like this.
udp 0 9968 x.x.x.x:52911 y.y.y.y:53809 ESTABLISHED
But every second it dies off and respawns on some other pid. So, i am not able to kill it also.
Also, all the packets, cud be millions of them are being sent to the same IP y.y.y.y
Can anyone help me find the program which triggers these packets and stop it?
netstat -p
will give you the process responsible for the UDP packages.will tell you exactly what is going on.
Alternatively, fuser(1) or lsof(1) can tell you as well.