I've been banging my head off NRPE and Perl for weeks now.
I decided to start from absolute first principles, and create a dummy nagios plugin that does nothing but always returns OK. I called it check_true.pl, installed it on the remote server and configured NRPE to serve it out as check_test.
The entire script is just:
#!/usr/bin/perl
print "OK - this dummy test always returns OK\n";
exit 0;
This works over NRPE without problems.
With this as a starting point, I was going to build up the script I want slowly to see at what point it breaks. I didn't get at all far. The follow breaks over NRPE (but works fine with locally and over SSH):
#!/usr/bin/perl
use strict;
print "OK - this dummy test always returns OK\n";
exit 0;
It gives the dreaded error: NRPE: Unable to read output.
I can't include ANYTHING or I get this error. This makes it impossible to do what I actually need to do!
I thought it might be a problem with the perl include path, but running the following over NRPE shows it is not (gives identical include path as when run on the terminal):
#!/usr/bin/perl
print "OK - Perl include path: ".join(q{, }, @INC)."\n";
exit 0;
Does anyone know why NRPE is behaving so badly with Perl? Can anyone recommend a fix? Or even a workaround?
Update 1: The command is defined in NRPE as follows:
command[check_test]=/usr/lib64/nagios/plugins/check_true.pl
Update 2: I've done some more debugging, and by adding the wrapper below around the perl script I was able to capture STDERR.
#!/bin/sh
out=`/usr/lib64/nagios/plugins/check_true.pl 2>&1`
echo $out
The result makes things even more confusing:
Can't locate strict.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/lib64/nagios/plugins/check_true.pl line 3. BEGIN failed--compilation aborted at /usr/lib64/nagios/plugins/check_true.pl line 3.
A quick search shows that strict.pm is located at /usr/share/perl5/strict.pm, and /usr/share/perl5 IS in the listed @INC!
How can Perl fail to find a file that is right there? This works flawlessly when run as the user nagios in a terminal, so what is NRPE doing to the environment to mess up Perl?
The culprit was SELinux which on RHEL6 constrains the NRPE process greatly. By default it even blocks access to the core perl modules like strict and warnings.
I haven't yet found a nice solution for turning off SELinux for just NRPE.