I know this is a very common error where RTFM is usually the correct answer, but, I think I've covered all the obvious points.
NRPE is running as the user nagios
When I use sudo su - nagios
to become the nagios user, and copy and paste in the command from /etc/nagios/nrpe.conf it works perfectly, generating output as expected.
When I turn on debugging, nothing is logged after the NRPE plugin loads (there are entries logged while the service is restarting).
What can be going on? Surely something should be logged in the debug logs? Is there a way to force more verbose logging (comments in config file suggest not).
SELinux is not logging any issues.
In the general the script is quite normal, with one exception, it shells out to a command using sudo. The nagios user has been granted passwordless access to the command being shelled out to, and that it working when I su to the nagios user.
Update: After re-configuring sudo so it does not insist on a tty, I have my check working successfully over SSH, returning the expected result. It still will not work over NRPE though.
The problem was SELinux which, on RHEL at least, stops the NRPE service doing pretty much anything.
The simplest solution is to turn off SELinux, but of course that is not ideal. The only other solution seems to be to write your own custom policy for NRPE, which is beyond my ability ATM.