Home / server / Questions / 925878
In Process
sam
Asked: 2018-08-11 10:36:34 +0800 CST

How to diagnose false positives with a smart PDU ping'ing devices on LAN

  • 772

Ive got a smart PDU (Pakedge) which pings, various LAN devices in the rack and if they dont respond it power cycles the associated power outlet to reboot the device, it also sends me an email alert.

What safe guards can i put in place to stop false positives coming up where for instnace the network switch is powered down and then every device would "look offline" / frozen to the PDU ?

UPDATE Some answers have correctly mentioned that force power cycling devices in the above manor could cause issues, i should clarify the device that im doing this to in this above instance are rack mounted AV matrixes / amplifiers rather than servers / NAS

power-distribution-unit

2 Answers

  1. Do not do this. You have already identified a potential problem if a switch goes down and the PDU power cycles other devices. And power cycling has risks for the integrity of some systems that should be shut down gracefully.

    Instead, design the high availability that you need.

    • Define your uptime requirements.
    • Monitor the services these devices support from the end user perspective. Maybe for a web server, GET the login page and track every http status code.
    • When the service availability is inadequate, find the root cause of the outage.
    • Where a single component failed, you can start adding redundancy. Hot spare routers, load balancers, clusters, and so on.

    There are cluster implementations that "shoot nodes in the head" by power cycling them. Corosync + Pacemaker, aka Red Hat cluster suite, can do this. But they have an idea of quorum, and only do so when most of the nodes agree it is dead. And a good cluster implementation requires testing to be sure it reliably fails over, and only when necessary.

    • 2

  2. Is the network switch manageable ?

    If yes, you can think about these solution.

    Step 1: ping the switch. If it is running, continue with the other checks. Otherwise do nothing, and start againt to step 1.

    If the switch is not pingable, you can for example ping the mail server or another host on the network.

    Beware there are pros and cons adding additional checks. You risk in certain conditions to not power cycle the LAN devices.

    • 0