Recognize rootkit-taken server

First thing to ask is for what reasons the intruder has your server in the first place. The motivation for illegal activities is most likely money. It probably isn't anything that needs calculation power, so it is not that important to monitor the usage of system resources.

So, how to use hijacked computer for profit? By stealing information or turning it into a spam server etc. All of these has in common that they need network connections. So you should monitor for abnormal network traffic.

As the compromised server may hide these connections (making them invisible from the system itself) the most effective way is to have an external firewall with an Intrusion Prevention System between the server and the internet.

Many kernel level rootkits can be effectively squashed at attack time by enabling the kernel command line module.sig_enforce.

http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt?v=4.8

 module.sig_enforce
                 [KNL] When CONFIG_MODULE_SIG is set, this means that
                 modules without (valid) signatures will fail to load.
                 Note that if CONFIG_MODULE_SIG_FORCE is set, that
                 is always true, so this option does nothing.

In order to take advantage of this properly you'd ideally want a 'vanilla' kernel thats only using modules that come from the main repository for the system though. The upshot of doing this is you only allow to load kernel modules which are signed by your trusted repository you originally got your kernel from.

In regards to detection and/or prevention of userspace rootkits -- rpm -V can help provide some anomalies out of the box but if the ssl libraries have been tampered with this is still possible to outdo.

Ultimately these types of security checks are fruitless though. If you are interested in maintaining some degree of tamper resistance for your systems you need to alter your mindset of how security works.

You need to define what the system should do and not what the system should not do. Often this is both difficult to enumerate in terms of time or unknown to the system administrator which is the crux of the problem.

However, once you know what the system should do you can define security policies in things like SELinux (a whole massive subject on its own, but turning it on is a great start). SELinux these days has pretty good policies which have gone a long way to trying to define what a typical Linux should should do already.

A future step (if you are really really seriously paranoid) is you buy hardware which contain TPM chips and setup servers to be fully tamper resistant via the IMA facilities being merged into the kernel.

This relies on a physically tamper-resistant chip which provides crypto-services to the kernel, through which you can cryptographically sign specific files and binaries in order to make this truly tamper resistant (the kernel can refuse to execute code that isn't signed in this way).

Most of this is so fresh and new that its not really distro ready unfortunately, but its good to know of its existence for when it does become ready.

https://wiki.gentoo.org/wiki/Integrity_Measurement_Architecture

There are all kinds of malware out there, and the well written kinds will be very hard to detect. Most malware these days is written with the intent to use your server to target other people (other than ransomware, which is unfortunately easy to detect), so the best external places to look revolve around what your server is sending out. Three of the more common goals of malware are to make money via phishing sites, spam, and DDOS services.

Phishing sites can generally be detected via your webserver's access logs. The thing to look for is a randomly generated subdirectory somewhere within a site's document root, or a directory made to look like part of your application framework (I've seen wp-files and wp-config for WordPress for example). Inside this folder will be a page that looks like the login page of a popular service or financial institution, as well as either a text file or a mail client to get the user's information. This requires some knowledge of what your access logs generally look like, but is fairly easy to spot.

Malware that causes a server to send spam can usually be detected by checking your mail queue, as it is highly likely that there will be a number of bounceback errors. You can also use a tool like senderbase to see if there has been any abnormal spikes in your mail traffic. Blacklist checkers like MxToolbox and multirbl are useful as well.

Detecting if your server is participating in DDOS attacks can be more difficult to detect. The best way is to keep track of your estimated bandwidth usage, as sudden spikes in your outbound is fairly indicative of malware.

Overall, your list is a good start as malware can manifest is many different forms. You will just have to keep in mind that well written malware will be all but impossible to detect without using an IDS/IPS and other monitoring software.