A couple of days ago our clients reported our Solarwinds website was down. So I connected to check through remote desktop and there it was, a fullscreen html-like interface where you could only use the mouse to type using the interface buttons.
Fullscreen html-like interface
Our team first action was to isolate the cluster from the network and then locally connect to each member on site to see if all got hijacked. To our surprise, all servers from the cluster of 4 were compromised.
I started making tests on our Web and DB server, while my teammates were formatting the least important members of the cluster like the poller servers which hold close to none critical data for the operation.
I was able to get all our data back from the DB and Web server by booting from a Fedora live CD/USB and accesing Windows' filesystem (of course it was not encrypted as they claimed) then went into System32 folder and found their script, then changed its main folder name so it could crash (I decided to only change the name in order to later on try to analyze it and gather some information from it).
I have spent a couple of hours looking at it, but most of the code was compiled into an .exe (they also used some .bat and .ps1 scripts, so those I have already seen). I opened the .exe as plain text and found what it seems to be Python comments from libraries and some other text (including the html-like interface message) I think the code is written in Cython and then compiled the .c file thrown by the cython setup file but I am not sure if I'm right or if there is a possible way to decompile it and get as much code as possible.
And other thing that I don't know if it is possible is to see how could have they possibly got into the server, like which protocol, I have some dates where they created tasks in the Windows task scheduler, I also checked the event viewer logons but couldn't understand it since there are a lot of logons. I am not very skilled with Windows and my programming skills are medium level, maybe you know of other stuff I can check in the server that could provide valuable information. If you need me to attach the .zip containing most of what the hacker put into our server let me know, because I need to put some instructions in order for you to prevent to infect yourself. Btw, we have Windows Server 2016.
UPDATE 1: First of, thank you guys for the help and recommendations. I work in a very unorganized company, these servers didn't even have Window's Firewall enabled. The company does not have any kind of insurance, procedure or politics for what to do or how to act in this kind of situation, probably because they had been lucky enough that it hadn't happened before.
The guys up in the organization chart pacnicked, the first instruction was say nothing about what really happened and if questions are asked, answer as obscure as possible.
Of course my boss first reaction was to tell us we need to restore the service ASAP. So while I was working trying to find and break the hacker's ransom script in the web and db servers, 2 engineers were formatting the first 2 poller servers. When I found the solution it was already too late, the data of 2 servers were already lost. I convinced my boss to leave the web server just as it was, so I could check all the available logs, the code and anything else I could find.
Now that I've extensively reviewed all the logs in the web server (it took me 2 days) and readable pieces of the script (non-compiled code) I realized that the one server I took home was not the entry point of the hacker, I can only see sessions from other members of the cluster which were previously compromised so I was not able to determine how did the hacker get into the server in the first place or which server was compromised first. I was able to make assumptions that he gained access through some file transfer protocol, since I later found PsExec application installed manually by the hacker.
Also, I didn't think to get a copy of the volume and then format, next time I will have this in mind in order to keep all the evidence.