I have a VPS running Ngix and currently hosting a few websites. As you know VPS have low resource and the security measures should be done by client.
I just noticed many stress tools are out there which can cause a webserver crash or the server eats full resources which can end up hanging. I have got LoadUI in my windows pc. There are even online similar services too, like LoadImpact.com
It doesn't even need to run 10 or thousands tools at the same time, Even just a kid can enter the domain name in these tools and run the test with tons of concurrent connections and make full use of server bandwidth, hardware resources,etc..
I want to know How should I prevent these flooding attacks ? Is it something should be handled by Iptables ? Or Nginx ?
That you are already running nginx is a good start - event based servers are much more resilient against sloloris type attacks.
Still it's a good idea to prevent DOS attacks as far awayas possible from your application. The next step is iptables.
You need to think about how you clssify attacks and differentiate them from real traffic - the speed at which new conections are being created is a very good indicator - and you can configure iptables to limit new connections on a per ip basis:
(drops new connection requests when the rate rises above 80 each 30 seconds)
You can limit the number of concurrent connections per ip address:
It's also a good idea to limit bandwidth, depending on your traffic profile to, say10% of the available bandwidth - this is done using tc rather than iptables.
Then, for the connections which get through, there may be characteristics in the HTTP request which would identify an attack (referrer, URL requested, user-agent, accept-language....) it doesn't matter what specific values you pick for these just now - you just need to esure that you've got the machinery in place where you can quickly change the parameters at the first sign of an attack. While you could handle the request on the webserver, a better solution is to block access from the remote IP address using iptables - fail2ban is the tool for bridging your log data to your iptables config.
Of course for a large scale DDOS this isn't going to solve the problem of the attackers stuffing your internet pipe with packets your server ignores - for that you need to speak to your upstream provider.
2 thing that I would recommend looking into is iptables rate limiting and fail2ban. Fail2ban will give you some decent auto blocking of IPs that are hitting your server too much and allow you to customize how long you want them to be banned. Iptables rate limiting will allow you to throttle all types of traffic coming into your server. I found a decent article about it here. However if you do a basic Google search you will see a lot more.
Edit: While I have no personal experience with nginx, I do see that it has an HttpLimitReqModule that you should look into as well.