Thursday, January 16, 2020

DBVisit Daemon as a Cluster Resource

DBVisit 8 introduced the ability to run dbvisit standby in the background. On RAC configurations the dbvctl daemon will run only on one node (in previous version this could be scheduled on all instances via cron jobs). If the database instances on the node where dbvctl daemon is currently running shuts down then the daemon should be relocated to a node with a running DB instance. Below is the action script that could be used for this purpose. It's a modification of the dbvnet action script. It is assumed the new cluster resource will be named dbvctld.
#!/bin/bash
#
# DBVCTL Action Script


#  Function to change database environments
###########################################

## set following to ensure oraenv is picked up from /usr/local/bin
export PATH=/usr/local/bin:$PATH

set_env ()
{
  export ORAENV_ASK=NO
  export ORACLE_SID=$1
  . oraenv >> /dev/null
  export ORAENV_ASK=YES
}

#################
## Main Section
#################

# This is logged to CRSD agent log file
echo "`date` Action script '$_CRS_ACTION_SCRIPT' for resource [$_CRS_NAME] called for action $1"

# set environment
set_env std11g2
DDC=std11g2
cd /opt/dbvisit/standby


case "$1" in
     'start')
           ./dbvctl -d $DDC -D start
           RET=0
           echo "Running start dbvctl resource with return code $RET"
     ;;

     'stop')
           NUM=`ps -ef | grep dbvctl | egrep -v 'grep|action-script|resource' | wc -l`
           if [ $NUM = 0 ]; then
             ## do a cleanup of pid
             ./dbvctl -d $DDC -D stop
             RET=0
           else
             ## now stop the dbvctl
             ./dbvctl -d $DDC -D stop
       
             NUM=`ps -ef | grep dbvctl | egrep -v 'grep|dbvctld' | wc -l`
      
             if [ $NUM = 0 ]; then
               RET=0
             else
               RET=1
             fi
           fi
           echo "Running stop dbvctl resource with return code $RET"
     ;;

     'check')
           NUM=`ps -ef | grep dbvctl | egrep -v 'grep|action-script|resource|dbvctld' | wc -l`

           if [ $NUM = 0 ]; then
             ## return code 1 for check means OFFLINE
             RET=1
           else
             ## return code 0 for check means ONLINE
             RET=0
           fi
           echo "Running check dbvctl resource with return code $RET"
     ;;

     'clean')
           for c1 in `ps -ef|grep dbvctl |egrep -v 'grep|action-script|resource'| awk '{print $2}'` ;
           do
             echo "...force kill dbvctl pid $c1"
             kill -9 $c1
           done
           ## do some cleanup
           ./dbvctl -d $DDC -D stop
           RET=0
           echo "Running clean dbvctl resource with return code $RET"
     ;;

esac

if [ $RET -eq 0 ]; then
  exit 0
else
  exit 1
fi

Add the cluster resource specifying the action script.
crsctl add resource dbvctld -type cluster_resource -attr "ACTION_SCRIPT=/opt/dbvisit/standby/action-script.scr, RESTART_ATTEMPTS=3, START_TIMEOUT=60, STOP_TIMEOUT=60, CHECK_INTERVAL=10,START_DEPENDENCIES='hard(appsvip1,dbvnetd,dbvagentd,ora.std11g2.db) pullup(appsvip1,dbvnetd,dbvagentd,ora.std11g2.db)',STOP_DEPENDENCIES='hard(appsvip1,dbvnetd,dbvagentd,ora.std11g2.db)' ACL='owner:oracle:rwx,pgrp:oinstall:rwx,other::r--' PLACEMENT='favored' HOSTING_MEMBERS='rhel6m1'"




Check cluster resource failing over and dbvctl daemon starting when database instance goes down. The appsvip1 is the VIP used for dbvisit standby in the RAC configuration and dbvagentd and dbvnetd are cluster resource created for dbvagent and dbvnet.
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
appsvip1                            app.appvip.type           ONLINE             ONLINE             rhel6m1
dbvagentd                           cluster_resource          ONLINE             ONLINE             rhel6m1
dbvctld                             cluster_resource          ONLINE             ONLINE             rhel6m1
dbvnetd                             cluster_resource          ONLINE             ONLINE             rhel6m1
...
ora.std11g2.db                      ora.database.type         ONLINE             ONLINE             rhel6m1
ora.std11g2.db                      ora.database.type         ONLINE             ONLINE             rhel6m1
Stop the instance on node where dbvctl daemon is currently running. As there's a hard dependency between DB instance and dbvctl daemon -f option must be used to stop the DB instance.
srvctl stop instance -d std11g2 -i std11g21 -f
Check the cluster resource status.
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
appsvip1                            app.appvip.type           ONLINE             ONLINE             rhel6m2
dbvagentd                           cluster_resource          ONLINE             ONLINE             rhel6m2
dbvctld                             cluster_resource          ONLINE             ONLINE             rhel6m2
dbvnetd                             cluster_resource          ONLINE             ONLINE             rhel6m2
...
ora.std11g2.db                      ora.database.type         OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type         ONLINE             ONLINE             rhel6m2