What is this server doing?

A couple places to start:

• listening services (netstat) - this should, generally speaking, give you a decent idea of what's going on with the system.
• /root/.bash_history (or that of other users, if they didn't use root) - whatever's been going on on the console will, ideally, be related to the purpose of the system.
• /var/log - take a glance at the standard logs, and look for anything application related.
• Installed packages - this is specific to the distribution of linux that they're running, but if the logs are there, take a look. /var/log/dpkg.log, /var/log/yum.log, etc.

Hardly scientific I know but if you get permission from your management I'd consider pausing the VMs - you'll find out if they're important quicker that you'd think, if it stays paused with nobody complaining...well that tells you something else.

Seriously though you could spend a career trying to figure them out without every truly knowing everything they do. Pausing them may seem odd/draconian but in the absence of documentation I'm sure you could sell the idea to management, as a one-off at first to see how it goes anyway.

I was surprised to see that the first answer suggested wasn't ps -ef, so I'll add it: if you want to know what a system is doing right now, read the process list, paying particular attention to what root is up to, and whether there are processes owned by conspicuously-named users (mysql, named, etc).

I'd then compare my process list against lsof run as root to see which processes are listening on the network, and which are holding open files. Typically this gives you a pretty good picture of the long-running processes on the box, which are commonly its main function.

Notable exceptions include mail - see local syslog and mailq for details on what's being processed by sendmail - and inted-type run-on-demand services, for which /etc/xinetd.conf is a good bet, at least for most recent Redhat-based Linuxes.

Hope that helps; let us know if you run into something in particular we can help identify!

I would start by seeing what services are running... Then attempt to match those to what they are hosting. Do NOT under any circumstance power off what you have no idea it's doing as you could break what ever it is running an if its mission critical( if that is the route your dying to take, pause them)... You should also check to see if there any sort of documentation.

Oh dear, that's a fun one.

Do you have any idea what they're used for? Can you narrow it down to "these were used for network services", or could it really be anything at all?

I would say a packet capture on each server is necessary, along with an audit of all running services. Locate the config files for each running service and check when the files were last updated - that will give you a clue as to whether something has been customized, and if so, how long ago.

You can also run a port scan on each server to see which ports are open and responding.

You can get clues by querying known network services - EG, DNS, LDAP, etc. You should be able to find a list of all DNS servers for a particular zone by digging for NS records. Bear in mind that you may end up with a longer list of NS records than there are actually active DNS servers, but it'll give you a starting point.

None of these methods are sure fire by themselves, but if you throw multiple audit methods at a particular box your chances of finding everything worth finding is enhanced.

Good luck!

A port scan would reveal any network accessible services

From the server locally: nmap 127.0.0.1

Or you can tell nmap to scan a certain subnet/mask

One other angle is look at what is configured to connect to the servers. If foozle.example.com is configured in the CEO's email client, it is probably the mail server. FTP clients probably point towards a web server of some sort. Etc, etc.

ps -ef for processes, netstat -a for services listening and tcpdump to see what traffic is going back and forth are great suggestions. In addition, since it's Linux, there's a good chance there's a firewall running - check out the rules set-up for it, should give you a good clue what services are expected to be used on this host and remote hosts that this host connects to. e.g. iptables --list Of course, what firewall is there is another thing to be checked out, try lsmod to look for firewall modules and check out /var/log