Home / server / Questions / 343057
Accepted
Crazy_Bash
Asked: 2011-12-22 02:45:58 +0800 CST

Netstat: How to know if it's Human or Bot/Spider/DDOS

  • 772

I'm using the following command to check how many connections I have from a single ip

netstat -anp |grep ':80' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

If it's DDOS attack(500+ connections), it's easy to detect. But I still have at any given time 60-100 connections from single Ip. Can it be a Human or it's some sort of bots/spiders?

This is what i have at the moment(top 6).

48 217.212.230.***
54 46.63.105.***
55 62.235.175.***
56 79.235.188.***
60 178.27.93.***
63 31.16.96.***
linux security netstat

4 Answers

  1. Best Answer

    Obviously this is difficult to answer empirically given we don't know what your service/application is or how it behaves - this number of connections could be by design for instance.

    But the best way to understand this issue is to benchmark typical site access from a single known IP, watch what happens during the 'user story' as you make your way through whatever transaction/searches you would normally expect a user to do.

    Now look at how the connections work. Does this benchmark match what you're seeing via netstat?

    If so then at least you know where you are. If they're substantially less, and there's no reason to assume those IPs have a valid reason to be the source of multiple 'user stories', then you probably do have some form of spider/DOS/whatever.

    Don't forget that you can always nslookup and geolocate these addresses. It can be a very useful tool when dealing with this sort of thing. Also don't be afraid to browse/probe them if you're concerned too.

    • 3

  2. You will need to dig deeper in the what is going on to see if the traffic is legitimate.

    Legit Traffic

    Legitimate bots can be identified by their user agents. Checking the IP's owner or reverse DNS can often help pinpoint spoofed user agent data.

    I have seen Googlebot, Yahoo's Slurp and other spiders create 40+ connections to a system.

    Note that browsers can make up to 20 connections (I think the default is around 8) to a server. If you have several users behind a NAT, they could easily use up 20-40 connections.

    Signs of DDoS

    I have seen DDoS attacks us a low level of connections but from a wide range of IP addresses. For example, the attackers will typically have no more than 100 connections to the server but from dozens or even hundreds of IPs.

    This tactic is used to undermine certain rate limiting methods available to block HTTP floods.

    In these cases, I take the top offending IPs and look at several items:

    • what country do they originate (IP Whois)
    • do they come from the same class C/B network
    • what are they accessing (check apache logs)
    • do they appear to have normal user behavior
    • commonalities in user-agent strings

    For example, during a recent DDoS mitigation, we identified that the platform doing the attack identified itself as a specific browser/language combination. Providing a rule to drop these requests eliminated 60% of the attack.

    In another case, I discovered that about 50% of the attacking IPs were assigned to one region in China. Using country specific IP blocks, we were able to drop this traffic.

    So digging into the top IPs should quickly tell you if you should allow it or block it.

    • 2

  3. I notice that you do not have an answer yet (at least you did not when I posted this on Security SE), so here's my two bits.

    A single browser (run by a human) should never send in that many connections. However, it is possible that many users in the same network on the same site could cause the number of concurrent connections to multiply. If they are on the same NAT network, they will use the same IP.

    You may wish to use a net inspector in a browser, test loading of your web pages, and gauge how many connections will be placed. The browser does have its own concurrent connection limit, but that is browser specific.

    A bot or a spider such as for a search engine will not likely make anything more than a browser.

    • 2

  4. Check the web server logs. The agent's name is there. All big bots identify itselfs.

    • -2