Hidden Features of Linux

To get the ball going, I find screen to be essential:

When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the users terminal.

Maybe I don't use these every day, but I use them frequently:

• strace Check out what files are loaded by the process.
• htop A nicer top.
• mtr ping + traceroute combined
• lynx/links/w3m In case you need console browsing
• ettercap Great network sniffer (i prefer it over wireshark)
• scripting bash Every *nix admin should know this.
• A programming language. For the more complex things, stay away from bash scripting and use something like python/perl/ruby/tcl/... (I use Lisp)
• Midnight Commander can be great for people who liked norton commander.
• irssi You never know when you just want to go ask something on IRC.
• wget / curl Download stuff from the command line.
• scp Copy stuff over ssh
• lftp / ncftp Good (scriptable) console FTP clients.
• iotop Check what's stressing your disks
• nmap good port scanner

lsof often gets ignored, its a very useful tool. lsof lets you view a list of every open file on the system, who / what is using it, etc.

For instance:

root@tower:~ # umount /mnt/hardy
umount: /mnt/hardy: device is busy
umount: /mnt/hardy: device is busy
root@tower:~ # lsof | grep /mnt/hardy
bash       5966       root  cwd       DIR      253,2     1024          2 /mnt/hardy
root@tower:~ #

Now I see that I'm logged into a shell in another terminal, and /mnt/hardy is my current working directory. So I can either kill that shell, or go to the other terminal and get out of that directory so it can be unmounted.

That's really a trivial example, its very handy for cleaning up the occasional 'bot' infestation too. The options are extensive, see man lsof for more.

I was surprised to find that you can run remote GUI applications over SSH, using the "-X" parameter. For example:

# on my machine
$ ssh -X linuxserver
# on remote machine
$ gedit /etc/my.cnf &

The gedit window appears on my local machine, editing the "my.cnf" file on the server.

I'm assuming this only works if your client machine has an X environment -- in other words, not on Windows. But it works great on my Mac!

To change to the last directory you were in:

cd -

Network stack can be left running after a system halt. I don't know if this is current with the 2.6.x series of kernels, but on older versions, you could configure the firewall/routing, then halt the system without a shutdown, leaving just the network stack running. This would allow you to make a cheap (although static) firewall that "can't be hacked" - because there's nothing to hack, as there are no programs or services, just the network stack portion of the kernel passing packets back and forth...

I like "locate" - a much easier way to find files than the gnarly "find . -name xxxx -print". Note that you have to use the updatedb command with it to a your file index up to date; see the man pages for details.

I find "ngrep" really useful for debugging network code on remote servers without having to punt tcpdump files around:

ngrep -d any -W byline port 80

for example, will show you live HTTP requests and responses.

One other thing I've found useful frequently is the "-e" switch to strace:

strace -p <pid> -e trace=open

will show all open() syscalls for a given pid, and

strace -p <pid> -e trace=\!rt_sigprocmask

will exclude all calls to rt_sigprocmask() from the output (useful for debugging RoR code, which seems to make an awful lot of those calls when built to use pthreads).

apropos

DESCRIPTION Each manual page has a short description available within it. apropos searches the descriptions for instances of keyword.

gyaresu@debian:~/bin$ apropos ettercap
etter.conf (5)       - Ettercap configuration file
ettercap (8)         - (unknown subject)
ettercap_curses (8)  - (unknown subject)
ettercap_plugins (8) - (unknown subject)

Thankfully I've only needed a couple of times, but the Magic SysRq key still remains one of my all time favorite hidden features.

Alt+SysRq+RSEIUB