Early signs of a bad sysadmin

Take this first part with a grain of salt, because it is perhaps influenced by my having worked as a contractor for so many years.

Consider looking at a contractor if your ability to pay is such that you can't attract top talent in a full-time capacity. If you're paying too little and asking for too much you're going to either get poorly skilled employees, employees who have glaring defects that may not be skill-related (poor interpersonal skills, substance abuse issues, etc), or you'll end up with a "revolving door" position where employees work for awhile and leave for better pay.

If your company is hung-up on both paying too little and needing someone for a given period of time, as opposed to fulfill a given set of tasks, then you're probably in a hopeless scenario. Likewise, if the tasks will keep a full-time employee busy and the company is planning on paying too little, then it's also hopeless. You will get what you pay for in the long run, one way or another.

My guess is that you don't really have a full-time need, and the company could probably spend the planned salary, or less, on a contractor who would do everything you need.

A contractor is much easier to "get rid of" if the relationship is a "bad fit". A contractor can typically be much more flexible than a full-time employee re: work logistics (weekends, evenings, etc). A good contractor is going to treat your company's needs with a very high degree of skill and care because they know how easily your company can sever the relationship and look elsewhere.

This is going to sound really trite, but more than any of the other items below, pay attention to your sysadmin's ability to communicate with others. Basic writing and speaking skills are important, and do a lot to indicate the state of the mental processes occurring "behind the scenes". A sysadmin's work should involve communication with other IT and non-IT employees, and an ability to communicate effectively is essential. Having an ability to form analogies and communicate abstract concepts is certainly nice "icing on the cake", but if your sysadmin can't even write complete sentences or speak complete thoughts then it's hopeless already.

There are points in everybody else's answers that ring true to me re: a "bad fit" (be it an employee or a contractor). I've been the guy who helps companies bridge the gap between firing a bad sysadmin and hiring a replacement, and I've seen a number of bad scenarios play out. (Being the person who is changing passwords, looking for "back doors", etc, while the sysadmin is over in the CEO's office being fired is fun work, but stressful, too.)

Some "IT specific" nasty attitudes I've seen (cribbing from some parts of other posters' answers here, unashamedly) in dysfunctional situations include:

• Rip everything out and start over: It's one thing to identify something that's a "ticking time bomb" and take care of it, but often in IT I run into (often immature and entry-level) sysadmins who seek to "build an empire" in their image, and obsess over removing old infrastructure for the sake of installing new. It's one thing to make a business case supported by facts and ROI projections, but I've seen this particular dysfunction as being nothing more than a strong personal drive to replace systems for the sake of replacement.

• I can't tell you that: These are the sysadmins who, while taking a strong personal ownership stake in their work, go too far and become overly possessive, secretive, and paranoid. The computers belong to the business, not the sysadmin. Failure to document work, disclose passwords, or be open about how systems work (or fail) isn't a good sign. I've heard some sysadmins cite "security" as a reason to be secretive, but security by obscurity isn't security. I've also heard sysadmins with this attitude say things like "Yeah, but if I give the passwords to so-and-so they'll just screw it up." Usually, this is accompanied by a veiled or overt statement of fear for being blamed if something goes wrong subsequent to the disclosure. If a business is so unstable that this fear is justified the sysadmin would do better to leave and find another job than to play games with secrecy.

• Blame somebody / everybody / anybody else: These are the sysadmins that constantly cite third parties, their predecessor, or the users experiencing issues as the cause of problems. Certainly, there are issues caused by all these factors, but a pattern of consistent and repeated finger-pointing is a bad sign. We've all had to deal with hardware errata, software bugs, and users creating problems for themselves. Being able to identify one of these sources as a root cause to an issue doesn't make it finger-pointing. Being unwilling to investigate an issue and identify a root cause, though, combined with the reaction of vaguely waving hands and saying "It's gotta be that buggy Windows / Linux / Cisco router / etc..." is cause for concern.

• Power trip: These are the sysadmins who delight and setting up roadblocks for users because of a personal agenda or a perceived business agenda. Again, it's one thing to place limitations on users for justifiable business reasons. It's quite another, though, to be the "preventer of IT services" simply for the mad power rush of being able to control others. I've seen this particular dysfunction extend into really nasty things like "e-stalking" of employees by reading their email, covertly performing screen / session captures, listening to phone calls, and just generally being a "creepy" person to others.

• Policies don't apply to me: Often combined with the "power trip" attitude, these are the sysadmins who refuse to be subject to the IT policies that they, themselves, otherwise enforce or dictate. While it can be benign and harmless, I've seen this cause nasty situations like threatened sexual harassment litigation (a sysadmin surfing and prominently displaying work-inappropriate content). Sysadmins are in trusted positions, and need to maintain an attitude of professionalism. Part of that attitude means playing by the same rules and being accountable like everyone else. Just because we have the ability to perform activities "off the record" w/ our elevated access permissions and rights doesn't mean that we should do it.

• Can't admit weakness: It takes a strong person to say "I don't know the answer to that, but I can find it for you." Everyone has gaps in their knowledge and experience. This particular dysfunction often results in situations where a sysadmin ends up vastly over their head. It's important to take calculated risks in career development, and it can be said that great personal growth occurs when people "bite off more than they can chew" and succeed. On the other hand, great expense (or outright failure) for a business could easily occur when a sysadmin decides to tackle important issues like disaster recovery or IT security and fails for lack of ability. Managers who unreasonably disallow their employees access to third-party resources / training / support can help to create this kind of culture. No one should be penalized for admitting that they don't know how to do something while expressing a willingness to help find the right answer (or, even better, learning how to do it themselves).

• These are my toys: This is the sysadmin who treats the business IT infrastructure as an exciting toy. It's one thing to identify a particularly interesting technology that happens to fulfill a business need well, but it's quite another to influence a business to spend money on technology for the unstated purpose of being something fun to play with. I've seen situations where sysadmins became enamored with a given technology and decides to bring that technology in to solve a problem not because it's suited to the business need, but because it's something they'd like to play with. I've seen this happen all kinds of things: fiber optics, virtualization, SAN gear, wireless networking, etc. Management should keep this in check as much as possible, but non-technical managers may always understand if a given technology really is something the business needs or not.

• I've always done it this way: This is the sysadmin who is dead set in their ways. Usually, I've found this combined with an attitude of "I don't want to learn about new things", too. Our field is changing. Some of the work that we did 10 years ago is automated today, and some of it remains the "same old, same old". Everything about our industry is constantly being revised, updated, and refreshed. Best practices change more slowly, but even they change too. It's unreasonable to expect that every sysadmin will keep up with the "cutting edge" of technology, but it's also unacceptable for a sysadmin to languish in years-old technology showing no sign of interest in updating skills. If a business is a growing concern, its IT operation should be forward-looking. (Obviously, there's a balance here, too. You can tip the scales too far and end up in a "these are my toys" scenario, as well...)

• No understanding of business: Business "does IT" because it helps in doing business efficiently. Any other use of IT in business is counter-productive. Too often I've seen sysadmins who are unaware of basic concepts of accounting and business (revenue less expenses equals profit, etc). I would never expect a sysadmin to be an expert in accounting, but I would expect them to understand the basic way in which a business incurs expenses for the purpose of turning a profit. In poor economic times, especially, it's nice to have your sysadmin understand where the money comes from and why the business makes the decisions it does related to where the money goes. A sysadmin who believes that IT stands apart from the "business" part of the business isn't an asset.

• No desire for continuity: In today's occupational culture, it's should be assumed that we'll all work for a variety of employers. Our job today isn't, statistically, going to be our job forever. A good sysadmin should prepare documentation not because "they might get hit by a bus", but because their eventual replacement will need it. An unwillingness to prepare documentation because of perceived "job security" reeks, to me, of an individual who has no desire for upward mobility. I don't work for a single employer anymore, but if I did I'd be planning for what I was going to be doing next, and keeping documentation up to date so that my replacement will have better time of it (just like I'd like from my predecessor at my next job).

Openness. You want to be able to see what he's doing and how he is doing it.

I would say the number one symptom of a trainwreck in progress is if the guy locks down everything and forbids anyone else to have access to the systems.

He may give all sorts of "Security" related warnings about allowing other people to have access and accounts and root privs on other machines, but often that is a smokescreen for someone who wants to look important and put your junk in a vice. It's easy to manage access in such a way as to allow access but maintain the security and accountability of a system.

Oddly, people do better work when they know it is going to be seen by someone else...

Some excellent answers so far; I'd like to add:

Being afraid of hard and/or dirty work. One shouldn't go out of one's way to invite hard and/or dirty work on oneself, of course, but when some nasty job needs doing it's a good sign if the person shows a willingness to roll up their sleeves and muck in.

Failure to realise that the reason they do their job is for the customers. Ultimately this is what it's all about; people need to be able to come in each morining, log on, and get their stuff. An admin who doesn't keep this in the back of their mind is failing in their job.

Letting themselves get out of touch with the people. It's easy to get suckered into thinking that you're on some ivory tower, and that you don't need to deal with users or answer calls. Users are a valuable and important source of feedback, and an opportunity to learn if something you've put in place is working well or not. Arranging to spend some time every month working with the helpdesk is great.

Being too much of a "by the book" person. OK, there are a lot of perfectly good and documented ways of doing things, so this one is most definitely NOT a case of one extreme or the other. I mean the type of person who clings to their MCSE manuals and treats everything in there as if it was the One, True and Only way.

Failure to take a proactive approach. A good admin will always be anticipating potential sources of trouble and dealing with them before they become a problem. A bad admin will just sit back and coast along letting things slowly disintegrate around them until the dreaded day when something collapses during office hours at a critical time.

Being a technology evangelist. I mean the type of person who would try to force in their favourite OS, apps or platforms irrespectively. You say you have SQL Server (which means you're a Windows house), so be on the lookout for someone who is constantly extoling the virtues of Linux or Lotus Domino, for example.

Forgetting to cover the basic stuff. It's a fairly huge field, and in order to be good at the complex fiddly stuff one needs a decent grounding in the basics. A good person will be almost immediately asking you about things like your backup strategy, your central documentaion repository, if you have a standard PC image, when the last time you had your firewall health-checked was, and so on. These are the things that keep you ticking over from day to day, and are equally as important as anything else.

I'd say that the two most important things to look for in a good sysadmin is structure in their work and a thirst for knowledge - so an absence of one or both of these would be an early warning sign.

Almost nobody will be able to walk in a do everything on day one but if you have the time for them to pick things up then don't focus on a lack of any particular skill/experience, if they're a good sysadmin they'll be researching the bits they don't know within minutes of walking in the door and will be up to speed quickly.

They should also be interested in what 'reference/test' systems/tools they have - this will show that they want to try new things without risking the production environment, they may want TOO much of this kit but better that they want it all than none at all.

Oh and consider using http://jobs.serverfault.com/ to find someone ok ;)

Chopper3 and damorg make very good points. In addition I would stress giving the new sysadmin time to settle in and get comfortable in both the position and the company. There's a human aspect that needs to be considered as it's generally akward and nerve wracking being the "new guy". They'll need time to "figure out" what you've got, how it's configured, etc., etc. and they'll need time to start feeling comfortable with the people and culture of the company. Don't be hasty to evaluate or make judgements about skill or personality traits that you see in them that might actually be a result of nervousness, etc.

Documentation of work. And some more documentation of work.

Edit: That came out wrong, but you get the idea. That's what a good sysadmin does, so that you can check in on his/her work.

When a problem occurs, in either production or a testing environment, does this person investigate the root cause, or assume it was a one-time incident?

Since this person won't have all the answers, does he or she have the interpersonal skills and modesty to seek help from others?

As @Chopper3 said, a thirst for knowledge.

Early signs of a bad sysadmin....

1. Sleeps in Server room
2. Comes out of the server room saying 'Please tell me we have good backups!'

Will add more as I think of them.

I'd like to add something, its a type of admin. Usually entry level and inexperienced.

I call them shotgun upgrader

Every now and then in the update cycles systems that worked stops and several hours, somtimes days, are lost. The shotgun upgrader has struck again. A good sysadmin should know the dependencies that are needed for your production system to run and and not break it everytime an upgrade has a potential to do this. I caught one in the act once.

He was in the process of doing an "unattended" dist upgrade of one of our debian systems. aptitude -y dist-upgrade > /dev/null 2>&1 (Its horrible, dont ever try it, it most likely wont boot again)

I asked, what are you doing? He replied redirecting to /dev/null, it clogs up the screen!

As mentioned by Chopper3, evidence of a structured, disciplined approach and willingness to learn are good signs.

On the flip-side, early signs of poor skillset or "fit" can include lack of patience with questions, unwillingness to explain technical reasoning, constant and aggressive defensiveness, never-ending finger-pointing at colleagues and/or predecessors (if there's merit to this, there's no reason to flog it to death over and over).

Also, a desire to "rip it all out" or redo everything "the right way" are tendencies to watch.

A certain amount of "well I would have done it this way" can be natural but unless there is also an assessment of the current environment and its weaknesses, and a reasonable plan to correct those problems and meet whatever other requirements there might be, and lots of discussion, I'd be wary.