Home / server / Questions / 470755
Accepted
Soviero
Asked: 2013-01-20 20:35:22 +0800 CST

Log all commands run by admins on production servers

  • 772

It is company policy for admins to login to the servers via a personal username, and then run sudo -i to become root. Upon running sudo -i, sudo will create an environmental variable called SUDO_USER, which contains the original user's username.

Is there a way to log ALL commands within syslog with something akin to the following syntax:

${TIME/DATE STAMP}: [${REAL_USER}|${SUDO_USER}]: ${CMD}

An example entry would be:

Sat Jan 19 22:28:46 CST 2013: [root|ksoviero]: yum install random-pkg

Obviously it doesn't have to be exactly the above syntax, it just has to include a minimum of the real user (eg. root), the sudo user (eg. ksoviero), and the full command that was run (eg. yum install random-pkg).

I've already tried snoopy, but it did not include the SUDO_USER variable.

centos logging bash sudo

7 Answers

  1. Best Answer

    Update: 2 more things that have popped up in the comments and in follow-up questions:

    • Using auditd this way will dramatically increase your log volume, especially if the system is heavily in use via commandline. Adjust your log retention policy.
    • Auditd logs on the host where they are created are just as secure as other files on the same box. Forward your logs to a remote log collection server like ELK or Graylog to preserve your logs' integrity. Plus, adding to the point above, it allows to more aggressively delete old logs.

    As was suggested by Michael Hampton, auditd is the correct tool for the job here.

    I tested this on an Ubuntu 12.10 installation, so your mileage may vary on other systems.

    • Install auditd:

      apt-get install auditd

    • Add these 2 lines to /etc/audit/audit.rules:

      -a exit,always -F arch=b64 -F euid=0 -S execve
-a exit,always -F arch=b32 -F euid=0 -S execve

    These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.

    • To get rid of auid=4294967295 messages in logs, add audit=1 to the kernel's cmdline (by editing /etc/default/grub)

    • Place the line

      session required pam_loginuid.so

    in all PAM config files that are relevant to login (/etc/pam.d/{login,kdm,sshd}), but not in the files that are relevant to su or sudo. This will allow auditd to get the calling user's uid correctly when calling sudo or su.

    • Restart your system now.

    • Let's login and run some commands:

        $ id -u
    1000
    $ sudo ls /
    bin  boot  data  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  scratch  selinux  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old
    $ sudo su -
    # ls /etc
    [...]

    This will yield something like this in /var/log/audit/auditd.log:

    ----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.239:576): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.239:576): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.239:576):  cwd="/home/user"
type=EXECVE msg=audit(1359968226.239:576): argc=2 a0="ls" a1="/"
type=SYSCALL msg=audit(1359968226.239:576): arch=c000003e syscall=59 success=yes exit=0 a0=10cfc48 a1=10d07c8 a2=10d5750 a3=7fff2eb2d1f0 items=2 ppid=26569 pid=26570 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)
----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.231:575): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.231:575): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.231:575):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968226.231:575): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968226.231:575): argc=3 a0="sudo" a1="ls" a2="/"
type=SYSCALL msg=audit(1359968226.231:575): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26569 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.523:578): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.523:578): item=0 name="/bin/su" inode=44 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.523:578):  cwd="/home/user"
type=EXECVE msg=audit(1359968229.523:578): argc=2 a0="su" a1="-"
type=SYSCALL msg=audit(1359968229.523:578): arch=c000003e syscall=59 success=yes exit=0 a0=1ceec48 a1=1cef7c8 a2=1cf4750 a3=7fff083bd920 items=2 ppid=26611 pid=26612 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="su" exe="/bin/su" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.519:577): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.519:577): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.519:577):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968229.519:577): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968229.519:577): argc=3 a0="sudo" a1="su" a2="-"
type=SYSCALL msg=audit(1359968229.519:577): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26611 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.543:585): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.543:585): item=0 name="/bin/bash" inode=6941 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.543:585):  cwd="/root"
type=EXECVE msg=audit(1359968229.543:585): argc=1 a0="-su"
type=SYSCALL msg=audit(1359968229.543:585): arch=c000003e syscall=59 success=yes exit=0 a0=13695a0 a1=7fffce08a3e0 a2=135a030 a3=7fffce08c200 items=2 ppid=26612 pid=26622 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="bash" exe="/bin/bash" key=(null)
----
time->Mon Feb  4 09:57:11 2013
type=PATH msg=audit(1359968231.663:594): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968231.663:594): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968231.663:594):  cwd="/root"
type=EXECVE msg=audit(1359968231.663:594): argc=3 a0="ls" a1="--color=auto" a2="/etc"
type=SYSCALL msg=audit(1359968231.663:594): arch=c000003e syscall=59 success=yes exit=0 a0=7fff8c709950 a1=7f91a12149d8 a2=1194c50 a3=7fff8c709510 items=2 ppid=26622 pid=26661 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)

    The auid column contains the calling user's uid, which allows you filter for commands run by this user with

     ausearch -ua 1000

    This will even list commands the user ran as root.

    Sources:

    • 103

  2. Remember that sudo itself logs all sudo commands in the syslog, so all priv'd users should be educated to not simply sudo to get a root shell but to :

    sudo command p1 p2 ... pn

    The problem with this or any approach I have thought of is that as the root user, it is quite difficult to prevent a user from evading any specific type of logging. Thus anything thing you try will be < 100% I am sorry to say.

    Education, documentation, enforcement and above all trust is what is necessary.

    • 11

  3. I was once faced with the same problem and had to come up with a quick and dirty solution - each sudo user will have their own history file once they run the command sudo -i

    In /root/.bashrc I added the following line -

     export HISTFILE=/root/.bash_history-$SUDO_USER
 export HISTTIMEFORMAT="%F %T "

    So every user who sudos to root will have a history file .bash_history-username.

    Another method -

    Add the following code to /root/.bashrc and it will append the username, sudo-user, and the command to the log file, where ever the notice level is set, most likely /var/log/messages.

    function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${SUDO_USER}:${COMMAND}"
}
trap log2syslog DEBUG

    Credit to - http://backdrift.org/logging-bash-history-to-syslog-using-traps

    • 8

  4. A number of establishments actually prohibit the use of auditd because it is resource intensive and may result in an opportunity for denial of service attacks.

    One solution is to configure the latest Korn shell (ksh-93, see http://kornshell.com/ for details) to log all commands executed as root to a remote syslog server, and then to require by policy that, except in emergency situations, sysadmins log on with personal accounts and execute the enhanced Korn shell via sudo. Examination of the logs can detect when an admin launches another shell from the approved shell in order to cover their tracks, and the SA can then be educated as necessary.

    • 3

  5. Sudo has something called sudoreplay when enabled sessions is logged and can be replayed later, works similar as the script command that make a typescript of terminal session that later can be replayed with the scriptreplay command.

    • 3

  6. Not that there's anything wrong with any of the other answers thus far, but if you decide that sudo's logging via syslog is satisfactory, may I suggest a wrinkle: log it via the network to a remote audit host.

    That gets around the problem of "now I have become root, I can remove any trace of my malfeasance from the logs". You may now be root on the local box, but you can't call that log packet back from the network, and you (presumably) don't have root privileges on the remote audit host.

    I've been doing this with some of the networks I manage for years, and it has two other signal advantages:

    Firstly, there's one place on the network to go to check all syslogs, which allows much easier correlation of incidents, and so is a one-stop shop for investigations like "When juno was complaining that NFS server hera was not responding, was anyone else complaining about the same thing at the same time? If so, hera is likely to be the problem, let's see what she logged; if not, juno's network connection is more suspect, let's see what else juno logged at that time.".

    Secondly, syslog log rotation becomes easier: you don't keep copies of logs on local hosts for more than a few days, but you make sure the audit server has huge amounts of disc space, and keep all syslogs there for several years. Plus if, say, you want to write them to WORM media for eg forensic audit purposes, you only have to buy one WORM drive.

    • 2

  7. Since version 2.0.0, Snoopy is able to log arbitrary environmental variables. Here is a link to Snoopy's example configuration file, where you can find all about the %{env:NAME_OF_MY_ENV_VAR_I_WANT_LOGGED} syntax.

    However, recent contribution pointed out that logging the owner of the tty is a fairly effective and elegant answer to the question of "Who executed that command, as root?".

    Disclosure: I am the maintainer of Snoopy.

    Edit: Changed link to point directly to relevant documentation (apparently, certain people can't find it in the repo) + slight rewording.

    • 2