Switch on PAM debugging to Syslog

A couple of things for you to try:

Did you enable logging of debug messages in syslog?

cp /etc/syslog.conf /etc/syslog.conf.original
vi /etc/syslog.conf

Add the following line:

*.debug     /var/log/debug.log

Exit with :wq!.

touch /var/log/debug.log
service syslog restart

You can enable debugging for all modules like so:

touch /etc/pam_debug

OR you can enable debugging only for the modules you're interested in by adding "debug" to the end of the relevant lines in /etc/pam.d/system-auth or the other /etc/pam.d/* files:

login   auth    required    pam_unix.so debug

Then debugging messages should start appearing in /var/log/debug.log. Hope this helps you out!

At least on CentOS 6.4, /etc/pam_debug is NOT used.

If the pam_warn.so module is installed, you can get some logging output this way:

auth required pam_warn.so

success required pam_warn.so

etc. This module ensures that it will not interfere with the authentication process at any point, but it logs meaningful stuff via syslog.

Update

After examining the code and doing some compiling, I found that (1) it's possible to enable this debug mode through the source, and (2) a RHEL patch makes the feature nearly unusable (at least the pam_unix module) and (3) it's probably better to patch the code anyway.

To get this to work for RHEL, you can get the Linux-PAM ... src.rpm (for any 1.1 version) and change the spec file as follows:

• Find the line beginning with

  %configure \

and after it, add --enable-debug \

• Remove the line or comment-out the line (above the previous one) that begins with %patch2

Then build the rpm and install (with force, to overwrite existing packages). Now create the file /var/run/pam-debug.log:

install -m 622 /dev/null /var/run/pam-debug.log

If the file does not exist, debug output will be sent to stderr.

• This sending out to stderr is, in my opinion, stupid, and is what causes the patch conflict. You can change that behavior by going into the file libpam/include/security/_pam_macros.h and replacing the 4 lines of

  logfile = stderr;

with

return;

On build, you'll get warnings about unreachable statements, but they can be ignored. You can make this change in a sed script (and put it in the %prep section of the RPM after the patches)...

sed -i 's/logfile = stderr;$/return;/' libpam/include/security/_pam_macros.h

IF you do this little patch, you can restore the %patch2, as it should work again properly.

Debian and Ubuntu (and maybe other distros) have a special log file into which all pam output is logged:

/var/log/auth.log

I've been struggling with a pam related problem for a day and a half, finally found out about this log file, and saved myself from insanity.

Here's a sample of the contents of this file when things don't go as planned.

Jul 10 09:31:14 vagrant-ubuntu-trusty-64 pamtester: pam_userdb(vsftpd:auth): user_lookup: could not open database `/etc/vsftpd_users.db': No such file or directory
Jul 10 09:36:20 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log

Here's how it looks when it works:

Jul 10 09:47:00 vagrant-ubuntu-trusty-64 sshd[5222]: pam_unix(sshd:session): session closed for user vagrant
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: Accepted publickey for vagrant from 10.0.2.2 port 54652 ssh2: RSA dd:3b:b8:2e:85:04:06:e9:ab:ff:a8:0a:c0:04:6e:d6
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=0)
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session closed for user root
Jul 10 09:51:41 vagrant-ubuntu-trusty-64 pamtester: pam_userdb(vsftpd:auth): user 'foo' granted access
Jul 10 09:51:44 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Jul 10 09:51:44 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=0)

Note that none of the other possibilities for enabling pam debug logging worked for me.

I just happened to spend several hours trying to find out how to enable debug logs in PAM on CentOS 6.4. Although this question is for Debian, I will still write down how to do it on CentOS in the hope that other people don't have to put in the time that I already have.

As it ultimately turned out, enabling debug logs in the pam CentOS package is straightforward. The difficulty stems from the fact that it involves recompilation of the package. So, first find the SRPM (e.g. pam-1.1.1-13.el6.src.rpm) from here. Folks who don't know about compiling packages from SRPMs, can refer to the steps on setting up a RPM build environment.

Here is the main step. Open the spec file and add --enable-debug to the %build section in the configure invocation. Recompile! Reinstall the newly created package. Finally, create the file where debug logs will get written.

$ sudo touch /var/run/pam-debug.log

If you don't create the file then a lot logs will fly by at the terminal, which might not be very useful.

Asket... I really loved your post :) I was fighting with stuff like this for the last 15 hours... (I might have had a 30 min break though)

Somehow i got it working by doing all the stuff you did, which means i have a /etc/pam_debug and debug on pam entries. BUT as in my case i was struggling with pam_mysql, I had to put another verbose=1 after debug on my pam entries:

mailsystem:~# cat /etc/pam.d/smtp

auth required pam_mysql.so debug sqllog=1 verbose=1 user=mail passwd=imverysecret host=127.0.0.1 db=mail table=mailbox usercolumn=username passwdcolumn=password crypt=1 logtable=log_auth logmsgcolumn=msg logusercolumn=user logpidcolumn=pid loghostcolumn=host logtimecolumn=time

account sufficient pam_mysql.so debug sqllog=1 verbose=1 user=mail passwd=imverysecret host=127.0.0.1 db=mail table=mailbox usercolumn=username passwdcolumn=password crypt=1 logtable=log_auth logmsgcolumn=msg logusercolumn=user logpidcolumn=pid loghostcolumn=host logtimecolumn=times

That "sqllog" is just to write logs in the DB.

So maybe this helps you just a little bit.

We all hate PAM. Good luck!

Something screwed with /etc/securetty and depending on the tty (which is somewhat random) a login was rejected or not. REALLY NICE, phew!

Could you expand on that a little?

Per securetty's manpage:

the file contains the device names of terminal lines (one per line, without leading /dev/) on which root is allowed to login.

The behaviour you describe sounds quite a lot like securetty is working normally (assuming you are indeed logging on as root).

Some ssh logins worked, some not.

Here too, there may be non-PAM restrictions in place, so it may help to provide some insight into what your /etc/ssh/sshd_config looks like.

Notably, from your description:

• if you were trying to log in as root, and failing, then that could be because of this line being present in sshd_config: PermitRootLogin no
• if some users/groups work, and others not, one reason could be the use in sshd_config of AllowGroups or AllowUsers. A sample line might look like: AllowGroups users admins

It is of course entirely possible PAM is part of the issue, but your main 'symptoms' sound to me like they could be explained by other means.

Basically this information is present in other answers, but mostly hidden in lines of text.

One option is to pass debug to a module, that makes it more verbose but not much.

If you need more information, you need to build pam with --enable-debug. Either by building from source (autoreconf && ./configure --enable-debug ... && make install), or from a source package (deb, rpm, ...), which is OS-specific. As a result, if it has enough permissions, it creates /var/run/pam-debug.log and writes to it. Otherwise, it outputs to stderr. You might want to create /var/log/pam-debug.log in advance for it to have enough rights.

About /etc/pam_debug, it didn't work for me.