Is there any way to make a seasoned Linux syadmin productive without giving him full root access?
This question comes from a perspective of protecting intellectual property (IP), which in my case, is entirely code and/or configuration files (i.e. small digital files that are easily copied). Our secret sauce has made us more successful than our smallish size would suggest. Likewise, we are once-bitten, twice shy from a few former unscrupulous employees (not sysadmins) who tried to steal IP. Top management's position is basically, "We trust people, but out of self-interest, cannot afford the risk of giving any one person more access than they absolutely need to do their job."
On the developer side, it's relatively easy to partition workflows and access levels such that people can be productive but only see only what they need to see. Only the top people (actual company owners) have the ability to combine all the ingredients and create the special sauce.
But I haven't been able to come up with a good way to maintain this IP secrecy on the Linux admin side. We make extensive use of GPG for code and sensitive text files... but what's to stop an admin from (for example) su'ing to a user and hopping on their tmux or GNU Screen session and seeing what they're doing?
(We also have Internet access disabled everywhere that could possibly come into contact with sensitive information. But, nothing is perfect, and there could be holes open to clever sysadmins or mistakes on the network admin side. Or even good old USB. There are of course numerous other measures in place, but those are beyond the scope of this question.)
The best I can come up with is basically using personalized accounts with sudo, similar to what is described in Multiple Linux sysadmins working as root. Specifically: no one except the company owners would actually have direct root access. Other admins would have a personalized account and the ability to sudo into root. Furthermore, remote logging would be instituted, and the logs would go to a server only the company owners could access. Seeing logging turned off would set off some kind of alerts.
A clever sysadmin could probably still find some holes in this scheme. And that aside, it's still reactive rather than proactive. The problem with our IP is such that competitors could make use of it very quickly, and cause a lot of damage in very short order.
So still better would be a mechanism that limits what the admin can do. But I recognize that this is a delicate balance (particularly in the light of troubleshooting and fixing production issues that need to be resolved right now).
I can't help but wonder how other organizations with very sensitive data manage this issue? For example, military sysadmins: how do they manage servers and data without being able to see confidential information?
Edit: In the initial posting, I meant to preemptively address the "hiring practices" comments that are starting to surface. One, this is supposed to be a technical question, and hiring practices IMO tend more towards social questions. But, two, I'll say this: I believe we do everything that's reasonable for hiring people: interview with multiple people at the firm; background and reference checks; all employees sign numerous legal documents, including one that says they've read and understood our handbook which details IP concerns in detail. Now, it's out of the scope of this question/site, but if someone can propose "perfect" hiring practices that filter out 100% of the bad actors, I'm all ears. Facts are: (1) I don't believe there is such a perfect hiring process; (2) people change - today's angel could be tomorrow's devil; (3) attempted code theft appears to be somewhat routine in this industry.
Everything said so far here is good stuff but there is one 'easy' non technical way that helps negates a rogue sys admin - the four eyes principle which basically requires that two sysadmins be present for any elevated access.
EDIT: The two biggest items that I've seen in comments are discussing cost and the possibility of collusion. One of the biggest ways that I've considered to avoid both of those issues is with the use of a managed service company used only for verification of actions taken. Done properly the techs wouldn't know each other. Assuming the technical prowess that a MSP should have it would be easy enough to have a sign off of actions taken.. maybe even as simple as a yes/no to anything nefarious.
If people truly need admin access to a system then there is little you can do to restrict their activities on that box.
What the majority of organisations do is trust, but verify - you might give people access to parts of the system but you use named admin accounts (e.g. you don't give them direct access to root) and then audit their activities to a log they cannot interfere with.
There's a balancing act here; you might need to protect your systems but you do need to trust people to do their jobs too. If the company was formerly "bitten" by an unscrupulous employee then this might suggest that the companies hiring practices are poor in some way, and those practices were presumably created by the "top managers". Trust begins at home; what are they doing to fix their hiring choices?
What you are talking about is known as the "Evil Sysadmin" risk. The long and short of it is:
The combination of these things makes it essentially impossible to stop malicious action. Even auditing becomes hard, because you have no 'normal' to compare with. (And frankly - a broken system may well have broken auditing too).
There are bunch of mitigative steps:
syslogand event level auditing, to a relatively tamper-resistant system that they don't have privileged access to. But collecting it isn't enough, you need to monitor it - and frankly, there's a bunch of ways to 'steal' information that might not show up on an audit radar. (Poacher vs. gamekeepers)But pretty fundamentally - you need to accept that this is a trust thing, not a technical thing. Your sysadmins will always be potentially very dangerous to you, as a result of this perfect storm.
Without putting yourself into an insane technical mind twist to try and come up with a way to give a sysadmin power without giving them power(its likely doable, but would ultimately be flawed in some way).
From a business practice standpoint there is a set of simple solutions. Not cheap solution's, but simple.
You mentioned that the pieces of IP you are concerned about are divided and only people at the top have the power to see them. This is essentially your answer. You should have multiple admins, and NONE of them should be an admin on enough systems to put together the complete picture. You of course would need at least 2 or 3 admins for each piece, in case an admin is sick or in a car accident or something. Maybe even stagger them. say you have 4 admins, and 8 pieces of information. admin 1 can access systems that have piece 1 and 2, admin 2 can get to pieces 2 and 3, admin 3 can get to 3 and 4, and admin 4 can get to 4 and 1. Each system has a backup admin, but no admin is able to compromise the complete picture.
One technique the military uses as well is the limitation of moving data. In a sensitive area there may only be a single system that is capable of burning a disk, or using a USB flash drive, all other systems are restricted. And the ability to use that system is extremely limited and requires specific documented approval by higher ups before anyone is allowed to put any data on anything that could lead to information spillage. Along the same token, you ensure that network traffic between different systems is limited by hardware firewalls. Your network admins who control the fire walls have no access to the systems that they are routing, so they cant specifically get access to information, and your server/workstation admins ensure that all data to and from a system is configured to be encrypted, so that your network admins cant tap the network and gain access to the data.
All laptops/workstations should have encrypted hard drives, and each employee should have a personal locker they are required to lock the drives/laptops in at the end of the night to ensure that no one comes in early/leaves late and gains access to something they aren't supposed to.
Each server should in the very least be in its own locked rack, if not its own locked room, so that only the admins responsible for each server have access to it, since at the end of the day physical access trumps all.
Next there is a practice that can either hurt/help. Limited contracts. If you think you can pay enough to keep attracting new talent, the option of only keeping an each admin for a pre-determined set of time (IE 6 months, 1 year, 2 years) would allow you to limit how long someone would have to attempt to put together all the pieces of your IP.
My personal design would be something along the lines of... Split your data into however many pieces, lets say for the sake of having a number 8, you have 8 git servers, each with their own set of redundant hardware, each administrated by a different set of admins.
Encrypted hardrives for all workstations that will touch the IP. with a specific "project" directory on the drive that is the only directory users are allowed to put their projects in. At the end of each night they are required to sanatize their project directories with a secure deletion tool, then hard drives are removed and locked up(just to be safe).
Each bit of the project has a different admin assigned to it, so a user would only interact with the workstation admin they are assigned to, if their project assignment changes, their data is wiped, they are assigned a new admin. Their systems should not have burning capabilities and should be using a security program to prevent the use of USB flash drives to transfer data without authorization.
take from this what you will.
It would be similar to the challenge of hiring a janitor for a building. The janitor gets to have all the keys, can open any door, but the reason is that the janitor needs them to do the job. Same with system admins. Symmetrically one can think of this age old problem and look at ways trust is granted historically.
Although there's no clean-cut technical solution, the fact that there's none shouldn't be a reason that we don't try any, an aggregation of imperfect solutions can give somewhat great results.
A model where trust is earned:
Implement several levels of administrative powers:
Always create an environment where total access by one person is not possible:
Use the Two-man rule when doing high-level core changes:
Trust and verify:
Paperwork:
It is very very hard to secure hosts against those with administrative access. While tools like PowerBroker attempt to do this, the cost is adding both something else that can break AND adding barriers to attempts at fixing it. Your system availability WILL drop when you implement something like this so set that expectation early as the cost of protecting things.
Another possibility is to see if your app can run on disposable hosts via a cloud provider or in a locally hosted private cloud. When one breaks, instead of sending in an admin to fix it, you throw it away and auto-build a replacement. This will require quite a lot of work on the application side to make them that run in this model, but it can solve a lot of operational and security issues. If done poorly, it can create some significant security problems, so get experienced help if you go that route.
This is your baseline discussion: https://security.stackexchange.com/questions/7801/keeping-secrets-from-root-on-linux
You split your responsibility by having security engineers whose job it is to make system configurations and installs, but they get no credentials or access to machines in production. They also run your audit infrastructure.
You have production admins who receive the systems but don't have the keys to boot the machine without the SELinux policies being active. Security doesn't get the keys to decrypt sensitive data stored at rest on disk for when they get a broken machine pulled from service.
Make use of a centralized authentication system with strong auditing like Vault and make use of its crypto operations. Hand out Yubikey devices to make keys absolutely private and unreadable.
Machines are either wiped on breakage or handled by ops and security together, and if you feel the need executive oversight.
Admins by the nature of the job have access to everything. They can see every file in the file system with their admin credentials. So, you'll need a way to encrypt the files so that admins can't see it, but the files are still usable by the teams that should see it. Look into Vormetric Transparent Encryption (http://www.vormetric.com/products/transparent-encryption)
The way it would work is that it sits between the filesystem and the applications that access it. Management can create the policy that says "Only the httpd user, running the webserver daemon can see the files unencrypted". Then an admin with their root credentials can try to read the files and only get the encrypted version. But the web server and whatever tools it needs sees them unencrypted. It can even checksum the binary to make it harder for the admin to get around.
Of course you should enable auditing so that in the event an admin tries to access the files a message gets flagged and people know about it.
The only practical way is restricting who can do what with sudo. You could potentially also do most of what you want with selinux but it would probably take forever to figure out the correct configuration which may make it impractical.
Non-disclosure agreements. Hire a sysadmin, they have to sign an NDA, if they break their promise, take them to court. This may not prevent them from stealing secrets, but any damages they cause by doing it are recoverable in court.
Military and government sysadmins have to obtain security clearances of different grades depending on how sensitive the material is. The idea being that someone who can obtain a clearance is less likely to steal or cheat.
Another way how to lower risk (use next to those mentioned before, not instead of) is to pay good money to your sysadmins. Pay them so good that they don't want to steal from your IP and leave you.