labradort's questions

Nagios was upgraded from 3.5.1 to 4.0.8

I wanted to ask this in the nagios support forum, but an hour later, I do not receive a confirmation email for setting up my account...

nagios seems to run OK as a service, but the web CGIs are not working, and there are no errors in error.log for apache, nor nagios.log. I've checked permission, and looked at some of the C code having this embedded error:

Whoops! Error: Could not read host and service status information!

The same error above appears for almost every menu on the left side of the main page for nagios.

nagios.log looks like this on start and then stopping, from the init:

[1431102009] Nagios 4.0.8 starting... (PID=27779)
[1431102009] Local time is Fri May 08 13:20:09 ADT 2015
[1431102009] LOG VERSION: 2.0
[1431102009] qh: Socket '/usr/local/nagios/var/rw/query.sh' successfully initialized
[1431102009] qh: core query handler registered
[1431102009] nerd: Channel hostchecks registered successfully
[1431102009] nerd: Channel servicechecks registered successfully
[1431102009] nerd: Channel opathchecks registered successfully
[1431102009] nerd: Fully initialized and ready to rock!
[1431102009] wproc: Successfully registered manager as @wproc with query handler
[1431102009] wproc: Registry request: name=Core Worker 27785;pid=27785
[1431102009] wproc: Registry request: name=Core Worker 27786;pid=27786
[1431102009] wproc: Registry request: name=Core Worker 27782;pid=27782
[1431102009] wproc: Registry request: name=Core Worker 27781;pid=27781
[1431102009] wproc: Registry request: name=Core Worker 27783;pid=27783
[1431102009] wproc: Registry request: name=Core Worker 27784;pid=27784
[1431102009] Successfully launched command file worker with pid 27787
[1431102022] Caught SIGTERM, shutting down...
[1431102022] Successfully shutdown... (PID=27779)
[1431102022] Event broker module 'NERD' deinitialized successfully.

Running with -v is clean:

# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 816 services.
        Checked 826 hosts.
        Checked 11 host groups.
        Checked 0 service groups.
        Checked 18 contacts.
        Checked 13 contact groups.
        Checked 61 commands.
        Checked 6 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 826 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 6 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

Also, the check_nagios says we're running OK:

# /usr/local/nagios/libexec/check_nagios /var/log/nagios.log 5 '/usr/local/nagios/bin/nagios'
NAGIOS OK: 8 processes, status log updated 11 seconds ago

One possibility is the error means it can't access the nagios.cfg file. I've checked that the path to that is r-x for 'other' (to cover apache user) on all directories on the path. In any case, if there was a permission issue, that should make an apache error. I've been working on this for a couple of hours and can't find the point of failure, or what changed.

The main page also shows "Unable to get process status" under the Nagios Core logo. That is from running statusjson.cgi in main.php - not sure what it is looking at, but the page is blank when I run the CGI query (cgi-bin/statusjson.cgi?query=programstatus) from main.php manually. I've googled this, searched the nagios forums, but everyone else seems to have some log error(s) to give more clues.

I do have one anomaly...

I found another nagios.log which is being touched with just a couple of lines each time the service is started:

# cat /usr/local/nagios/var/nagios.log
[1431088940] Error: Cannot open main configuration file '/' for reading!
[1431088940] Error: Failed to process config file '/'. Aborting

Perhaps something wacky with the init or cfg files, but I can't find it. As another test, I can su to nagios and run the daemon manually.

su - nagios
[nagios@atlas ~]$ /usr/local/nagios/bin/nagios /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Nagios 4.0.8 starting... (PID=23234)
Local time is Fri May 08 13:45:12 ADT 2015
nerd: Channel hostchecks registered successfully
nerd: Channel servicechecks registered successfully
nerd: Channel opathchecks registered successfully
nerd: Fully initialized and ready to rock!
wproc: Successfully registered manager as @wproc with query handler
wproc: Registry request: name=Core Worker 23235;pid=23235
wproc: Registry request: name=Core Worker 23236;pid=23236
wproc: Registry request: name=Core Worker 23237;pid=23237
wproc: Registry request: name=Core Worker 23238;pid=23238
wproc: Registry request: name=Core Worker 23239;pid=23239
wproc: Registry request: name=Core Worker 23240;pid=23240
Successfully launched command file worker with pid 23241

I hoped this would avoid anything odd in the init script. It does not touch the /usr/local/nagios/var/nagios.log (expected), but it does not change the error from the web site cgis. Another clue is that when the nagios is started manually like this, I don't see any logging on the screen of hosts and status items. If I launch the init, there are some warnings on some host performance, flapping and the usual chatter from nagios logs, but it is not saying anymore than the above when started from command line as nagios user.

I can't seem to find an existing question like my scenario. I'm using 2 public bridges for 4 VMs. Two VMs per bridge device. Interface stats show there is use of the assigned bridge for the VM on inbound traffic, but outbound is going through only the first bridge.

# cat /etc/network/interfaces 

# The loopback network interface
auto lo
iface lo inet loopback

pre-up iptables-restore < /etc/firewall-rules

auto br0
iface br0 inet dhcp
    bridge_ports    eth0
    bridge_stp      off
    bridge_maxwait  0
    bridge_fd       0

auto br1
iface br1 inet dhcp
    bridge_ports    eth1
    bridge_stp      off
    bridge_maxwait  0
    bridge_fd       0


#  brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.a4badb4e3949       no              eth0
                                                        vnet0
                                                        vnet2
br1             8000.a4badb4e394a       no              eth1
                                                        vnet1
                                                        vnet3
# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vlan-200.mydoma 0.0.0.0         UG    0      0        0 br0
XXX.YYY.200.0   *               255.255.248.0   U     0      0        0 br0
XXX.YYY.200.0   *               255.255.248.0   U     0      0        0 br1

The routing makes sense with what I'm seeing: all outbound traffic, unless for the 200 subnet, will use br0.

How can it be configured so that the KVM guests on br1 have their outbound traffic truly use br1 for their gateway? The current set up works fine, but I'd prefer to assign the bridges to VMs as their own full gateway so br1 is better utilized for TX packets.

Edit: This is not a question about DDOS, but a question about how to resolve a technical issue impacting only Mac clients of Outlook Anywhere.

Problem and solution are are now known, but I cannot link it here for some stupid reason of stackexchange limits. I can suggest googling : "outlook anywhere We Get to Use Commandlets" and you should find the problem and solution described at charlietree.com

Original question follows...

I'm not a Mac user, nor the Windows Admin, so forgive me if I don't have the the nomenclature correct, but I'm trying to help another admin.

We run Active Directory and Exchange 2010. The name servers for our Internet top level domain are Linux with Bind. A subdomain like ad.example.com is the domain for AD, Exchange, etc.

In an attempt to prevent the DNS service on AD from being abused with DNS reflection DDOS attack method, port 53 was blocked at the firewall. It has the effect of blocking off site users with Mac Outlook from syncing with Exchange.

Blocking port 53 seemed the only way to go because disabling recursion on the Windows DNS causes failures to access the outside world, and unlike Bind, there is no feature like views.

Are other sites finding this is a problem, or does it hint of a configuration problem?

The admin mentioned that when traced, the connection information (perhaps with autodiscovery) came back with the address like exchange.ad.example.com, while the exchange server is also known as an address like exchange.example.com. He isn't sure if there is some place in the configuration to fix that. The idea being if we can get the "ad" out of the host name, the Mac Outlook client would not need to talk to the DNS on AD.

Our Goal: to block AD's DNS servers from DDOS abuse.

Our problem: Mac Outlook clients require access to AD's DNS when off site.

This is on Solaris 10 sparc platform.

I'm trying to make our denyhosts script block the dictionary ssh attacks. Unfortunately, the logging does not show an IP when the user name is invalid.

We are using winbind module in pam.conf

If I intentionally enter a bad password for a good user name, it logs like this:

Mar 30 14:49:21 t2000 sshd[29870]: [ID 800047 auth.notice] Failed keyboard-interactive for fpicabia from 111.222.333.444 port 52567 ssh2

This shows the IP which I want to keep for denyhosts to scan later. No problems here.

If I login with a bogus user name, the log shows no IP info:

Mar 30 14:55:57 t2000 sshd[1816]: [ID 186046 auth.error] pam_winbind(sshd-kbdint): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_USER_UNKNOWN (13), NTSTATUS: NT_STATUS_NO_SUCH_USER, Error message was: No such user

In /etc/ssh/sshd_config I already use verbose for logging:

SyslogFacility auth
LogLevel VERBOSE

In pam.conf I have these settings for winbind:

login   auth sufficient         pam_winbind.so.1 debug
other   auth sufficient         pam_winbind.so.1 try_first_pass debug

On Linux, the winbind module reports on the IP in the auth log with only the Loglevel of INFO and no debug turned on at the pam module. The limitation is specific to Solaris and its winbind module.

How can I get the IP address showing in the authentication log when the user is unknown?

I have a Postfix mail server running Debian Squeeze. SASL authentication for secure SMTP is supported by saslauthd which relies on pam, which relies on winbind.

This pattern seems to be reoccuring every couple of weeks:

1. Initially, there are no errors in /var/log/auth.log - postfix SMTP is working well

2. Errors appear in the log after a week or more:

   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM unable to dlopen(/lib/security/pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: Too many open files Dec 5 15:45:22 myhostname saslauthd[32586]: PAM adding faulty module: /lib/security/pam_winbind.so
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM unable to dlopen(/lib/security/pam_deny.so): /lib/security/pam_deny.so: cannot open shared object file: Too many open files
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM adding faulty module: /lib/security/pam_deny.so
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM _pam_load_conf_file: unable to open /etc/pam.d/common-auth
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM error loading (null)
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM _pam_init_handlers: error reading /etc/pam.d/other
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM _pam_init_handlers: [Critical error - immediate abort]
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM error reading PAM configuration file
   Dec 5 15:45:22 myhostname saslauthd[32586]: PAM pam_start: failed to initialize handlers
   Dec 5 15:45:22 myhostname saslauthd[32586]: DEBUG: auth_pam: pam_start failed: Critical error - immediate abort
   Dec 5 15:45:22 myhostname saslauthd[32586]: do_auth : auth failure: [user=dteed] [service=smtp] [realm=] [mech=pam] [reason=PAM start error]
   Dec 5 15:45:32 myhostname saslauthd[32586]: server_exit : master exited: 32586
   Dec 5 15:45:32 myhostname saslauthd[1696]: detach_tty : master pid is: 1696
   Dec 5 15:45:32 myhostname saslauthd[1696]: ipc_init : listening on socket: /var/run/saslauthd/mux

At this time authentication will always fail. It can be verified in testsaslauthd. Simply restarting saslauthd corrects the problem for another week or two.

I've searched for the error on too many open files and there is nothing solid on what this indicates. Some of the pam modules it complains cannot be opened have nothing to do with the job saslauthd is handling of simply SASL authentication over winbind.

Here is the /etc/pam.d/smtp :

account     required    pam_permit.so
auth        sufficient    pam_winbind.so debug
auth        required      pam_deny.so

I don't want to run saslauthd in the debug (foreground) method. I don't think I need more saslauthd processes - I'm running 5 and this was sufficient on a previous lower powered server. The secure SMTP is an option which is not widely used by our users, but there is probably one connection every couple of minutes using it.

The pre-compiled packages of lsof for Solaris (e.g. from Blastwave or sunfreeware) do not show the ports open, such as when using the options : "-Pni". The wisdom of the creator of lsof is that we should compile lsof ourselves. The tar bundle, configuration, etc., are very different than most open source packages one can compile from source. I'm running low on clues within the package to learn what is wrong with my build attempt.

I have Sun Studio 11 to provide C compiler. Also installed Gcc 3.4.5

The system is sparc, with ZFS on all file systems. OS is Solaris 10, U7.

uname -a
SunOS myhostname 5.10 Generic_141444-09 sun4u sparc SUNW,Sun-Fire-280R

lsof source is for 4.82. I configure with default options and ZFS support on.

With either cc or gcc, the failure when running make is similar:

gcc  -Dsolaris=100000  -DHASPR_GWINDOWS -m64 -DHASIPv6 -DHASSOUXSOUA -DHAS_IPCLASSIFIER_H -DHAS_CRED_IMPL_H -DHAS_V_PATH -DHASZONES -DHAS_ZFS=1 -DZFS_VERS=15 -DHASUTMPX -DHAS_VSOCK -DHAS_STRFTIME -DLSOF_VSTR=\"5.10\"  -O  -c  dnode.c
dnode.c: In function `process_node':
dnode.c:2082: error: structure has no member named `pcfs_entps'
*** Error code 1
make: Fatal error: Command failed for target `dnode.o'

I did get this to compile on a non-ZFS system with 4.77 many months ago, but I can't remember how.