linux + create simple watch dog process in shell script

I'd not recommend trying to keep a process running all the time to do this. There are more simple methods. Your machine should have cron running which is a periodic task scheduler. You can schedule a process to run periodically, as frequently as once per minute to check the contents of the file and do what needs to be done. You might add something like this to the crontab:

* * * * * /path/to/yourscript

see man 1 crontab and man 5 crontab and man 8 cron for more information about cron.

Even better is to use incron which allows you to specify a process to be run any time this file is changed. If you have incron installed you'd add something like this to the incrontab:

/etc/cluster.cf IN_MODIFY /path/to/your/script

Saying that anytime /etc/cluster.cf is modified, run your script. see man 5 incrontab and man 1 incrontab

Assuming you are using a SysV distribution, create an initialization script and place it in /etc/init.d.

Look at any of the scripts that are already there for examples of how to format this script. Consider the ones that use the daemon function. You would then use chkconfig to enable the script to run at boot. This initialization script should write it's PID into a lock file. You will need a second 'helper process' to check for the first one's PID by reading the lock file and determine whether or not it is running. Include logic to destroy the lock and restart the first process if it does not find a running PID.

There are a thing whose only job is to (re)start other things, its name is init, and it is configured through inittab. To make something really immortal, add it to inittab with respawn option.

A simple checker script (a candidate for inittab) might be like this:

while :
do
     grep -q machineA_is_active /etc/cluster.cf && activation_script
     # here one needs to ensure the above wan't fire again
     # - say, by carelessly wiping off /etc/cluster.cf
     # or carefully editing out the triggering record
     sleep $delay
done

a very simple solution (not the most elegant) for a watchdog is to create a cron job that executes a service that checks for another service's status.

sudo crontab -e

then put this line at the end of your crontab file (attention this will run evry minute of every day until you change crontab again)

* * * * * /usr/sbin/sample_service

then CTRL-X, Y, ENTER and reboot your machine.

this is an example of such a service:

    #!/bin/bash
    #
    # watchdog
    #
    # Run as a cron job to keep an eye on what_to_monitor which should always
    # be running. Restart what_to_monitor and send notification as needed.
    #
    # This needs to be run as root or a user that can start system services.
    #
    # Revisions: 0.1 (20100506), 0.2 (20100507)

    NAME=sample_service
    NAME2=sample_service2
    START=/usr/sbin/$NAME
    START2=/usr/sbin/$NAME2
    [email protected]
    [email protected]
    GREP=/bin/grep
    PS=/bin/ps
    NOP=/bin/true
    DATE=/bin/date
    # MAIL=/bin/mail
    RM=/bin/rm

    $PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1
    case "$?" in
     0)
     # It is running in this case so we do nothing.
      echo "$NAME is RUNNING OK. Relax."

     $NOP
     ;;
     1)
     echo "$NAME is NOT RUNNING. Starting $NAME and sending notices."
     $START 2>&1 >/dev/null &
     NOTICE=/tmp/watchdog.txt
     echo "$NAME was not running and was started on `$DATE`" > $NOTICE
     # $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
     $RM -f $NOTICE
     ;;
    esac

     # GT06
    $PS -ef|$GREP -v grep|$GREP $NAME2 >/dev/null 2>&1
    case "$?" in
     0)
     # It is running in this case so we do nothing.
      echo "$NAME2 is RUNNING OK. Relax."

     $NOP
     ;;
     1)
     echo "$NAME2 is NOT RUNNING. Starting $NAME2 and sending notices."
     $START2 2>&1 >/dev/null &
     NOTICE=/tmp/watchdog.txt
     echo "$NAME2 was not running and was started on `$DATE`" > $NOTICE
     # $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
     $RM -f $NOTICE
     ;;
    esac

    exit