Home / server / Questions / 643616
In Process
Craig
Asked: 2014-11-12 12:33:06 +0800 CST

Best way to trace outgoing requests from a server

  • 772

We have an Ubuntu 14.04.1 LTS server with nginx 1.4.6 that hosts a few dozen websites and received an abuse complaint from another server of hundreds of requests like this: "POST /wp-login.php HTTP/1.0" coming from our server. How can I go about dissecting the problem so that I can find out where this traffic is coming from? I have been trying to read tcpdump output but I don't have much experience with it and and don't know what parameters to give it or how to read the output in such a way that I can find what I'm looking for in an efficient way. What tools should I be using and in what order to see if I can trace the source of the problem?

security

2 Answers

  1. A simple first level approach is to use lsof -i @remotehostname (or the remote ip-address) to see which user and process is initiating the connections to the remote host.

    With regards to interpreting tcpdump output, I still find it much easier to use something like Wireshark to help interpret a captured TCP stream. It also helps you generate the syntax to use in tcpdump to make your captures more specific.

    • 1

  2. Due to lack of information I'm going to guess following.

    • you system hosts web sites
    • web sites are vulnerable, probably some older versions with quite a few known vulnerabilities
    • websites hosted were likely automatically enumerated and exploited by one or more attackers
    • attackers have left backdoors on filesystem and possibly database as well
    • attackers didn't necessarily gain root access (the newer kernel you have the less likelihood they did)
    • using vulnerabilities and backdoors attackers are actively using your system to anonymize them selves while attacking other targets (as report you got confirms)

    What to do about it?

    • Keep web sites hosted up to date, easier said than done but this is important

    • Compartmentalize, your web sites should be isolated as much as possible from each other (php execution not as nobody or www-data but rather as unique users, by doing so compromise of one site will not lead to compromise of all sites as easy, especially when lazy attackers are giving you headaches)

    • scan and audit your system often, you can use Maldet to search for suspicious files, backdoors intruders may have left over. This is by no means complete cleanup suite, it will find some stuff but it's signature based and any mutation in backdoor can easily make backdoor go unnoticed by scanners like these.

    • search and destroy; co-relate; and by this I mean, search with whatever you have, make notes on findings, examine access logs for mentions of access to what you found, then note IPs that were accessing those bad files; then check which possibly other files did those IPs access on your server. This will probably reveal few more backdoors which signatures didn't match.

    • Try to reduce attack surface. Administration areas are feature rich components and this translates to a lot of bugs there. If you restrict access to administration areas to only handful of IPs; or add additional layer like basic auth it will help protect your hosted sites from getting owned.

    • Keep your eyes open, loop over this list, find new methods to add on this list to improve your defense capabilities.

    • 1