Specifically, I am asking about screwdrivers with removable tips (something like this), but I think the answer would apply to any magnetic-tipped screwdriver.
Will the magnetic field damage anything in the server?
I Googled this before asking, and the general consensus was "it won't hurt anything". However, the answers I found were from general PC guy forums, not the server community. Around high-dollar equipment, you certainly tend to be more careful!
I guess you might be able to affect a tape, but that's it. So don't store with your backup tapes. Hard drives have extremely powerful magnets for moving the heads (seriously, take one appart, immensely strong), and those are closer to the platter than your screwdriver can get.
So don't sweat it, aside from tapes and of course, floppies (if anyone remembers those).
Most modern computer electronics are shielded against magnetic fields, or are unaffected. If the magnet would require sufficient effort to remove from ferric (iron-based) materials that you might damage something else trying to yank it off... you're probably OK. A fairly weak magnetic screwdriver won't hurt anything.
Losing a screw in a case... can short out plenty of things or cause additional damage trying to extract the screw... or damage to components from ESD... and all of these significantly more likely.
I've used 'em dozens and dozens of times without issue.
I mean, don't rub them on the little sram or flash chips or anything, you'll be fine.
My method: Don't put them on top of HDDs, really near to streamer tapes or near a streamer write head. I don't think you would actually damage anything, but I tend to be extra careful when it comes to magnetic media.
Other than that, I can't see a problem and in fact, any of my screw drivers have magnetic tips, saves a lot of time in tight spaces sometimes.