Asad Moeen's questions

I am using Debian 6 - 64-bit and this is what my /etc/network/interfaces file looks like

auto eth0
iface eth0 inet static
    address PrimaryIP
    gateway X.X.X.X
    netmask 255.255.255.0
auto eth0:1
iface eth0:1 inet static
    address SecondaryIP
    gateway X.X.X.X
    netmask 255.255.255.0
auto lo
iface lo inet loopback

I need to switch between the Primary and Secondary IPs for most of my processes/scripts so quite a lot of times I swap PrimaryIP with SecondaryIP (and vice versa) in the above file and restart the interfaces to change the primary IP system is working on.

A problem I'm facing with this is that although the System's Primary IP is set in the above file and works perfectly after networking restart, the session/ssh IP still remains the secondary one. So where the system uses the PrimaryIP, anything I execute in the SSH session is still executed as the SecondaryIP.

Note: I have also tried logging in with both the IPs but the session still gets locked to the secondary IP although any system commands such as cronjobs are executed as primary.

I've been using tcpdump without any issues before this error popped up. I use the following two commands with variations as soon as my server is attacked so the packets automatically get dumped if the packet rate is high.

tcpdump -i eth0 -p -nn -s0 -c 2000 -w $dumpdir/dump.`date +"%Y%m%d-%H%M%S"`.cap
tcpdump -nn -s0 -c 2000 -w $dumpdir/dump.`date +"%Y%m%d-%H%M%S"`.cap

The following error shows up in console as soon as I'm under attack:

tcpdump: can't create rx ring on packet socket: Cannot allocate memory

Using the command manually works so my guess is that something gets overloaded in the event of an attack. Here is my RAM usage though:

root@x:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2004       1255        749          0          1         29
-/+ buffers/cache:       1225        779
Swap:         2047       1095        952

And currently, it is much lower than it used to be when the command worked.

So anyone with an answer to this?

I have been using my own self-made "iptables rules" for blocking all the major type of DDOS attacks on game-servers because these attacks were application/game-server specific instead of the general DDOS attacks. It took me almost 3 years to study those attacks and respond with these rules. Up til now, I was using a rate-limit on iptables that would automatically DROP the IP if it made 250 hitcounts/sec as the client rate is much lower than this (maximum it goes is like 500kb/s). This is because attackers would generally send a large number of hit-counts with packets of the length 15:30 as this is the general packet-length where the game-server responds well. This worked really well until someone attacked with something different today to get through this rule.

-A INPUT -p udp -m udp --dport 16000:29000 -m recent --set 
-A INPUT -p udp -m udp --dport 16000:29000 -m recent --update --seconds 1 --hitcount 250 -j DROP 

Recently, I observed attacks that were easily making 1-3mb/sec of input, as seen through "iftop" to my servers and the above rules weren't blocking them because my game-server was responding with "....disconnect" packets to all the incoming connections. This happens when game-server doesn't recognize the input string or length of the packet. This was the content of the packet:

http://paste.ubuntu.com/6000381/

Now as you might as well see that the packet had a huge length. According to Wireshark where I grabbed the packet, the length of the packet was 700 and data size/length was about 5000 bytes. And I've seen this before that if you have a higher data-length/size of the packet, you can still have a higher rate even with a fewer hitcounts. So this explains maybe why it had an input of around 3mb/s and it didn't get blocked through the hitcounts, because it never had hitcounts greater than 250..

Now I need something that would more be bandwidth specific than the number of packets/second. I require an iptables rule that would automatically block an input>1mb/s and it may use the length of the packet if needed. The general data-size/length of a fair game-server client would usually have packet lengths and data-size of less than 500 bytes but it might go higher in some cases but will never take more than 500kb/s of bandwidth.

The question is different than the rest of previously answered questions because it's firstly application specific. Secondly, I am asking an iptables solution that would deny any input >1mb/s?

Update:

I did some theoretical working to explain why a 3 Mb/s input bandwidth is not more than 250 hitcounts/second and here is the working:

3 Mb = 3x1024x1024 = 3,145,728 bits/sec = 393,216 Bytes/sec

Hence,

If a packet had the size of 5000 bytes so the maximum number of packets per second should be:

393216/5000 = 78.6 Packets/second

And this means that if the iptables rules had to work, the packet should have been smaller than 393216/250 = 1516 bytes.

I use iptables to block different kind of attacks on my server. We have a different set of rules along with different rate-limit rules. Now I also use a script that would observe the threshold if it were greater than 10mb/s and dump all the packets to a file. This script keeps running all the time in a screen session and uses the following command to dump:

tcpdump -nn -s0 -c 2000 -w Attack.cap
sleep 300

Once attacked, it waits 5 minutes to check for another attack (sleep 300). Now I doubt if during the packets capture process, iptables still work because in /var/messages I see lines like "eth0 entered promiscuous mode" and "eth0 left promiscuous mode" so it might over-look iptables?

I'm looking over a script that would actually be restarting game-servers in case they crash. The game-servers are actually started in a screen session through a control panel. They have to be restarted by killing their respective screen session and starting them again in a new screen session with the same name. I've done this script by my self but it doesn't really work and runs into errors.

#!/bin/bash

START='cd /home/test; ./GameServer +set parameters'
SCREEN_NAME="test"


while /bin/true; do


sleep 5
SERVER=`ps --User test | grep GameServer | grep -v grep | wc -l`

 if [[ $SERVER -eq "0" ]]; then
    screen -S $SCREEN_NAME-X quit ; screen -S $SCREEN_NAME -md $START
    sleep 10
 fi
done

The process name is "gameserver" and an issue is with killing the screen session because it won't quit and if I kill all screen sessions, the screen running the script also quits.

test@de:~$ screen -list
There is a screen on:
        31861.2-test    (08/11/13 12:51:06)     (Detached)
1 Socket in /var/run/screen/S-test.

test@de:~$ screen -r 2-test -X quit
No screen session found.
test@de:~$ screen -x 2-test -X quit
No screen session found.
test@de:~$ screen -S 2-test -X quit
No screen session found.

I can although attach to the above screen by "screen -x" and I can also kill a screen with the same name using the above commands if I start it as: "screen -S test". I believe there's something in the Control panel that starts it in a different way such that it fails to attach using the name.

Okay so I use the simple linux ftp client to download/upload files off ftp servers. I have an iptables firewall which blocks most of the ports and I have to close the firewall for it to work. Although port 21 is open but I guess that explains why the download works and not the upload. The following command works perfectly with the firewall on:

wget ftp://user:[email protected]

I just have problems when I actually connect to the server and try to "put"/upload a file to the server. Here is an example netstat output of the port used at that time but it's always different.

netstat -a | grep ServerIP
tcp        0 197520 myIP.:59622 ServerIP:ftp-data ESTABLISHED
tcp        0      0 myIP.:40341 ServerIP:ftp      ESTABLISHED

Both the ports on myIP range are blocked and I'm failing to guess what ports do I have to open. Google search fails as well. Secondly, if I try something like this on iptables, it gives me an error:

-A INPUT -p tcp --match multiport --dport 40000:40500 -j ACCEPT
iptables-restore v1.4.8: too many ports specified

Secondly, why would I need to open the ports when I have the following line before the ports were blocked in the configuration file:

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

I am using a Debian 6 - 64bit OS and my server is ddosed/flooded through udp protocol from time to time. I basically host game-servers and my firewall is set to rate-limit the packets on allowed ports but sometimes the rate per IP is low but the number of IPs are more so my server sends outgoing icmp unreachable replies to the attacking IP which does no benefit but chokes/saturates the port even more.

I'm looking on how to disable this feature. Actually the ports that get attacked are allowed through firewall and I can't disable them although it would solve the issue. I run a number of servers over some port ranges so I can't continually keep accepting these ports one by one and decided to allow the total port range I might require.

I'm looking at some kernel ability to stop this?

I'm using this script that actually checks for the rate of incoming packets and it gets triggered if the rate hits 5mbps or more. The packets are then logged to a tcpdump file.

interface=eth0
dumpdir=/tmp/

while /bin/true; do
  pkt_old=`grep $interface: /proc/net/dev | cut -d :  -f2 | awk '{ print $2 }'`
  sleep 1
  pkt_new=`grep $interface: /proc/net/dev | cut -d :  -f2 | awk '{ print $2 }'`

  pkt=$(( $pkt_new - $pkt_old ))
  echo -ne "\r$pkt packets/s\033[0K"

  if [ $pkt -gt 5000 ]; then
    echo -e "\n`date` Under attack, dumping packets."
    tcpdump -n -s0 -c 2000 -w $dumpdir/dump.`date +"%Y%m%d-%H%M%S"`.cap
    echo "`date` Packets dumped, sleeping now."
    sleep 300
  fi
done

The output is something like 2000 Packets captured. XXX packets recieved by Filter and XXX-(minus)2000 dropped by Kernel.

Now what I want to know here is that the output file wouldn't actually tell me the rate of the attack like if it was 300mbps or what? So is the XXX packets recieved by filter is per second? If not, how do I check that because my port sometimes gets saturated.

UPDATE:

I used a program to capture statistics from the captured file through the above script. Here is what I got:

root@$:/tmp/dumps# capinfos dump.20130621-174506.cap
File name:           dump.20130621-174506.cap
File type:           Wireshark/tcpdump/... - libpcap
File encapsulation:  Linux cooked-mode capture
Number of packets:   2000
File size:           2065933 bytes
Data size:           2033909 bytes
Capture duration:    43 seconds
Start time:          Fri Jun 21 17:45:06 2013
End time:            Fri Jun 21 17:45:49 2013
Data byte rate:      46968.49 bytes/sec
Data bit rate:       375747.94 bits/sec
Average packet size: 1016.95 bytes
Average packet rate: 46.19 packets/sec

I believe the attack might only have lasted a good 15-20 seconds while the captured information was 43 seconds so the data-bit rate here might have been averaged over from this total time. What might help here is if someone could edit the original script above instead of capturing 2000 packets and dropping the rest, to capture all packets for a duration of lets say 5 seconds when the threshold hits.

UPDATE:

After changing the script as mentioned, it looked like the file was damaged as I read it in Wireshark which said "The capture file appears to have been cut short in the middle of a packet." Here is output from capinfos:

capinfos: An error occurred after reading 3085 packets from `"dump.20130710-215413.cap": Less data was read than was expected.

On a 2nd try, I was able to read that file only when I pressed Ctrl+C in console of the script:

capinfos dump.20130710-215413.cap
File name:           dump.20130710-215413.cap
File type:           Wireshark/tcpdump/... - libpcap
File encapsulation:  Linux cooked-mode capture
Number of packets:   18136
File size:           2600821 bytes
Data size:           2310621 bytes
Capture duration:    591 seconds
Start time:          Wed Jul 10 21:54:13 2013
End time:            Wed Jul 10 22:04:04 2013
Data byte rate:      3909.73 bytes/sec
Data bit rate:       31277.83 bits/sec
Average packet size: 127.41 bytes
Average packet rate: 30.69 packets/sec

Notice capture duration 591 seconds. I believe the 'sleep 300' has something to do over here because as I see the console output. This output is with '-c 2000' option:

./Log.sh
10275 packets/s
Wed Jul 10 12:41:31 MSD 2013 Under attack, dumping packets.
tcpdump: listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
2000 packets captured
100012 packets received by filter
98003 packets dropped by kernel
Wed Jul 10 12:42:34 MSD 2013 Packets dumped, sleeping now.

Now this is the output after you modified the script with 'sleep 5':

./Log.sh
24103 packets/s
Wed Jul 10 21:54:13 MSD 2013 Under attack, dumping packets.
tcpdump: listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
Wed Jul 10 21:54:18 MSD 2013 Packets dumped, sleeping now.
1620 packets/sroot@nl:~# 18136 packets captured
1850288 packets received by filter
1832106 packets dropped by kernel
^C

Notice I pressed Ctrl+C to break the sleep function I guess which made reading of the file possible.

So I'm running a server with iptables config and I've only been checking for destinations. Here is what the config structure is. Top of config has a list of ip addresses to drop followed by rate-limiting on specific destination ports then accepting the one's rate limited. In the end I blocked all the ports. So the one's not accepted are blocked.

Now I need to fix a source port match somewhere in the config. I want to block udp source port X so where would this come?

I placed it at the end of the config but I am not sure if it works. Would it check the destination of that source port first and allow it? or would it just block the source port.

UPDATE:

Here is what it sort of looks like.

# Block Source Port 53 - The commands below slow the SSH down
#-A INPUT -p udp --sport 53 -j DROP
#-A INPUT -p tcp --sport 53 -j DROP 

#
# Destinations
#

# Accept PPTPD VPN Ports 
// Commands go here

# Accept IP Rules and Port Range
// Commands go here

# Rate-Limit ICMP Ping Requests
// Commands go here - Used with the iptables recent module

# Rate-Limit Game Servers Ports
// Commands go here - Used with the iptables recent module

# Accept the Ports Rate-Limited above
// Commands go here

# Drop All Other TCP & UDP Ports
-A INPUT -p tcp -j DROP
-A INPUT -p udp -j DROP
-A INPUT -j DROP

I tried the source port 53 block at the end of the config where it didn't seem to work possibly because the destination match of that source came first. Now I put it on top of the config or even in the middle somewhere and it seems to work but my SSH/SFTP login seems awfully delayed, such that it even times-out sometimes after I enter the username although my own IP is accepted among the IP addresses accepted. I also just tried blocking source port 53 at UDP because that's what I actually need but it would still cause that delay. Not sure what UDP has to do with ssh/sftp. Maybe the delay is caused by the source port checking actually?

I have some Game Server services running on my Debian 6.0 64-bit Custom Kernel machine.

I lag on my currently running services whenever I install a new service.

Installing a new service just involves extracting the files to a user's directory although I do it at the lowest priority ( nice 19 and 1000kbps SCP limit ).

This time I took a "vmstat 3" output during the time I was installing new service and when the lag occurred. I'm not good at reading it good though but I guess these are some disk issues.

http://paste.ubuntu.com/1152249/

See line numbers 11-16 for the time when it happened.

UPDATE :

Here is

df -h

Output: http://paste.ubuntu.com/1152734/

And the 250G disk is the one in use with those operations and services.

I'm basically done using SSH2 with PHP. Some may already that while using it, the PHP code actually waits for all the listed commands to be executed in SSH and when everything is done, it then gives back the results. Where that is fine for the work I am doing, but I need some commands to be multi-threaded.

$cmd= MyCommand; echo $ssh->exec($cmd);

So I just want this to run in parallel 2 times. I googled some stuff but didn't get along with it. For a basic thing, I came across to this way posted by someone but it didn't work out for me.

for ($i = 0; $i < 2; $i += 1)
    {
        exec("php test_1.php $i > test.txt &");  
    //this will execute test_1.php and will leave this process executing in the background and will go to next iteration of the loop immediately without waiting the completion of the script in the test_1.php , $i  is passed as argument .
}

I tried to put it this way exec("echo $ssh->exec($cmd) $i > test.txt &"); in the loop but either it never entered the loop or the echo $ssh->exec failed. I don't really need a very neat multi-threading. Even a single second delay would do good, thank you.

Well up-til now I've considered iptables rate-limit commands with the "recent" module to work for each IP Address. For example rate-limit rule of 20k/s will trigger only if a single IP exceeds 20k/s rate and not if 4 different IPs exceed 5k/s rate. Please correct me if I considered this wrong as I've only used these rules for TCP/ UDP.

But today I tried similar rules for ICMP and applied 4/s Input/Output. But then on trying to ping-test from just-ping.com I could see packet loss on almost all IP Addresses. How could that happen because if it worked for each IP Address then it wouldn't be triggering the rule because I believe each IP from just-ping has a rate of probably 1/s.

I still think the first one is true because if it wasn't then my GameServer would block everyone if the combined rate ( in case of more connected players ) increased the threshold. This hasn't happened up til now so the ICMP thing really confused me.

Thank you.

UPDATE

Here is my config:

-A INPUT -p icmp -m recent --set --name PING --rsource 
-A INPUT -p icmp -m recent --update --seconds 1 --hitcount 4 --name PING --rsource -j DROP 

-A OUTPUT -p icmp -m recent --set --name PING2 --rsource 
-A OUTPUT -p icmp -m recent --update --seconds 1 --hitcount 4 --name PING2 --rsource -j DROP 

What I basically want is to to write all tcpdump captured packets to a file every 3 days. So basically tcpdump should run for 24 hours on day 1 and write the output to Day1.log and similar for Day2 and Day3. On the 4th day it should repeat and write the log to Day1 again. This is basically to check DDoS attempts on my server and to find out the type of attack including the attacker's IP as in the last 7 days my machines were DDoS'd and I expect it to happen again. I know its done by some cronjobs but I need the actual commands to put there?

I also want to know which IP made how much input in mb/sec maximum as I have a high traffic so it would almost take me 6 hours to keep searching those files for the attacker's IP. So is there anything in WireShark during the analysis of those files which might tell how much input in mb/s was made by an IP to my server? If not, how should I find that?

Edit: --------------------------------------------

You guys are free to post your ideas of countering this as well. All I need is to find the attacker's IP, the packet-data he sent and the input in mb/s made to my server. My clients do not make more than 300kb/s input so if we set a filter to capture more than 1mb/s input if made, we could capture that.

My GameServers had been under some UDP Floods due to which they generated outputs to the attacker which gave the GameServers some huge lags. Thanks to friends at ServerFault that upon different kind of testing, I was able to successfully block the attack. My question is actually something else but it is important to know how the GameServers reacted to the attack and if the machine kept stable or not:

300kb/s Input would cause GameServer to generate 2mb/s Output. So as the Input Rate kept increasing, output rate would reach so high that it would no longer be possible for the GameServer to control it and hence it would give a huge Lag until the attack is stopped. Usually the game server starts to lag when it sends out something greater than 5mb/s and under that is controllable. Theoretically, I was able to receive a 60mb/s output from my GameServer on inputting 10mb/s. Its just the way the GameServer works if not protected.

Now on some of my machines, only the GameServer under attack lagged and although the server was generating 60mb/s output, rest of the gameservers on other ports would run fine without lags on the same machine.

But there was another machine which also runs on a 100 MBPS Network port, even 1 mbps input ( and ZERO output because attack is blocked ) even on an unused port would give a constant yellow line ( on the Lag-o-Meter ) to all the clients on all GameServers indicating lag because that line is actually blue under normal conditions. It would remain the same even on 50mbps or 900mbps input. I tried contacting the host about it because I believe its the way their Network is bridged, but they can't help me about it. Anyone else knowing about such issues because if 900mbps input does not Saturate the port, how can 1mbps input lag the servers although port is not saturated and enough bandwidth is available?

I host a GameServer and some stupid guy just keeps attacking my Application server with random queries. I have found a solution to block it but he uses different kinds of queries so I have to block the new ones almost daily using iptables. I bought a test VPS to try and change those flood strings and attack my Dedi just to be sure if everything is now safe because I can't wait for him to attack everyday.

Now the thing is, he uses multiple spoofed IPs in the attack. While my test tool can do it, it looks like some Networking issues not letting it happen. Lets say the target IP is target.com and it has to be attacked by attacker.com while test.com will be the IP that will be used for generating the attack from attacker.com to target.com.

Now I do see on the test.com machine using iftop that attacker.com is flooding target.com but no such attack appears entering the target.com machine. The script for the application flood is the same being used by the attacker and on asking the developer of the script, he says you have to directly be connected to the internet without a firewall/nat/ router.

I'm not good in Networking so I'd like to ask how can we achieve it. Aren't the VPS machines already directly connected, if not how can we get such a machine cheaply for testing?

Well I'm actually trying to block a UDP Flood on the Application port because the string "getstatus" is causing my application to make large output due to a small input to the attacker's IP.

I installed PSAD firewall to do the job.

psad -S shows 3000,000 logged packets at the application port and top ports in Scan but does not block the IP of the attacker however other IP Addresses with small number of connections are dropped. I'm thinking that since output is also being made to the attacker, this is why its not getting blocked because iptables rate-limiting is also exactly doing the same thing and not blocking the IP where outgoing connection is also made.

Any guesses why it won't work?

After a detailed analysis, I collected these details.

I am under a UDP Flood which is more of application dependent. I run a Game-Server and an attacker is flooding me with "getstatus" query which makes the GameServer respond by making the replies to the query which cause output to the attacker's IP as high as 30mb/s and server lag.

Here are the packet details,

Packet starts with 4 bytes 0xff and then getstatus.
Theoretically, the packet is like "\xff\xff\xff\xffgetstatus "

Now that I've tried a lot of iptables variations like state and rate-limiting along side but those didn't work. Rate Limit works good but only when the Server is not started. As soon as the server starts, no iptables rule seems to block it.

Anyone else got more solutions? someone asked me to contact the provider and get it done at the Network/Router but that looks very odd and I believe they might not do it since that would also affect other clients.

Responding to all those answers, I'd say:

Firstly, its a VPS so they can't do it for me. Secondly, I don't care if something is coming in but since its application generated so there has to be a OS level solution to block the outgoing packets. At least the outgoing ones must be stopped.

Secondly, its not Ddos since just 400kb/s input generates 30mb/s output from my GameServer. That never happens in a D-dos. Asking the provider/hardware level solution should be used in that case but this one is different. And Yes, Banning his IP stops the flood of outgoing packets but he has many more IP-Addresses as he spoofs his original so I just need something to block him automatically.

Even tried a lot of Firewalls but as you know they are just front-ends to iptables so if something doesn't work on iptables, what would the firewalls do?

These were the rules I tried,

iptables -A INPUT -p udp -m state --state NEW -m recent --set --name DDOS --rsource
iptables -A INPUT -p udp -m state --state NEW -m recent --update --seconds 1 --hitcount 5 --name DDOS --rsource -j DROP

It works for the attacks on un-used ports but when the server is listening and responding to the incoming queries by the attacker, it never works.

Okay Tom.H, your rules were working when I modified them somehow like this:

iptables -A INPUT -p udp -m length --length 1:1024 -m recent --set --name XXXX --rsource
iptables -A INPUT -p udp -m string --string "xxxxxxxxxx" --algo bm --to 65535 -m recent --update --seconds 1 --hitcount 15 --name XXXX --rsource -j DROP

They worked for about 3 days very good where the string "xxxxxxxxx" would be rate-limited, blocked if someone flooded and also didn't affect the clients. But just today, I tried updating the chain to try to remove a previously blocked IP so for that I had to flush the chain and restore this rule ( iptables -X and iptables -F ), some clients were already connected to servers including me. So restoring the rules now would also block some of the clients string completely while some are not affected. So does this mean I need to restart the server or why else would this happen because the last time the rules were working, there was no one connected?

Well, I've been a victim of Ddos attacks and I really can't figure out how to avoid it. On some machines it works, while on some it doesn't or probably I'm doing it wrong in some way,

Attackers are using multiple IPs to attack my game server ports that make my game-server output 1mb/s to each IP Address, these are UDP reflective attacks technically. Here is how I see to protect them,

    iptables -A INPUT -p udp -m state --state NEW -m recent --set --name DDOS --rsource 

    iptables -A INPUT -p udp -m state --state NEW -m recent --update --seconds 1 --hitcount 5 --name DDOS --rsource -j DROP

Technically this would block every attacker on UDP ports. I check if the attacker is blocked using "iftop" or "tcpdump" when I see that the output to attacker's IP becomes 0, this confirms me that the protection is working.

On my CentOS 6 machine running iptables 1.4.7, I am using similar method to block it and its really weird that it works sometimes and the next hour it doesn't. After a lot of tries, I restarted my system and iptables, protection worked fine and attacker was blocked. I logged into my system today and he is attacking me again and although I restarted iptables, hes not getting blocked. I dont want to restart my system again and again to make it work so what could be the issue?

Also, some people say UDP is state-less but whatever it technically is - I've always used this command and it has worked before for me, why not now or why partially now?