Thursday, January 16, 2020

DBVisit Daemon as a Cluster Resource

DBVisit 8 introduced the ability to run dbvisit standby in the background. On RAC configurations the dbvctl daemon will run only on one node (in previous version this could be scheduled on all instances via cron jobs). If the database instances on the node where dbvctl daemon is currently running shuts down then the daemon should be relocated to a node with a running DB instance. Below is the action script that could be used for this purpose. It's a modification of the dbvnet action script. It is assumed the new cluster resource will be named dbvctld.
#!/bin/bash
#
# DBVCTL Action Script


#  Function to change database environments
###########################################

## set following to ensure oraenv is picked up from /usr/local/bin
export PATH=/usr/local/bin:$PATH

set_env ()
{
  export ORAENV_ASK=NO
  export ORACLE_SID=$1
  . oraenv >> /dev/null
  export ORAENV_ASK=YES
}

#################
## Main Section
#################

# This is logged to CRSD agent log file
echo "`date` Action script '$_CRS_ACTION_SCRIPT' for resource [$_CRS_NAME] called for action $1"

# set environment
set_env std11g2
DDC=std11g2
cd /opt/dbvisit/standby


case "$1" in
     'start')
           ./dbvctl -d $DDC -D start
           RET=0
           echo "Running start dbvctl resource with return code $RET"
     ;;

     'stop')
           NUM=`ps -ef | grep dbvctl | egrep -v 'grep|action-script|resource' | wc -l`
           if [ $NUM = 0 ]; then
             ## do a cleanup of pid
             ./dbvctl -d $DDC -D stop
             RET=0
           else
             ## now stop the dbvctl
             ./dbvctl -d $DDC -D stop
       
             NUM=`ps -ef | grep dbvctl | egrep -v 'grep|dbvctld' | wc -l`
      
             if [ $NUM = 0 ]; then
               RET=0
             else
               RET=1
             fi
           fi
           echo "Running stop dbvctl resource with return code $RET"
     ;;

     'check')
           NUM=`ps -ef | grep dbvctl | egrep -v 'grep|action-script|resource|dbvctld' | wc -l`

           if [ $NUM = 0 ]; then
             ## return code 1 for check means OFFLINE
             RET=1
           else
             ## return code 0 for check means ONLINE
             RET=0
           fi
           echo "Running check dbvctl resource with return code $RET"
     ;;

     'clean')
           for c1 in `ps -ef|grep dbvctl |egrep -v 'grep|action-script|resource'| awk '{print $2}'` ;
           do
             echo "...force kill dbvctl pid $c1"
             kill -9 $c1
           done
           ## do some cleanup
           ./dbvctl -d $DDC -D stop
           RET=0
           echo "Running clean dbvctl resource with return code $RET"
     ;;

esac

if [ $RET -eq 0 ]; then
  exit 0
else
  exit 1
fi

Add the cluster resource specifying the action script.
crsctl add resource dbvctld -type cluster_resource -attr "ACTION_SCRIPT=/opt/dbvisit/standby/action-script.scr, RESTART_ATTEMPTS=3, START_TIMEOUT=60, STOP_TIMEOUT=60, CHECK_INTERVAL=10,START_DEPENDENCIES='hard(appsvip1,dbvnetd,dbvagentd,ora.std11g2.db) pullup(appsvip1,dbvnetd,dbvagentd,ora.std11g2.db)',STOP_DEPENDENCIES='hard(appsvip1,dbvnetd,dbvagentd,ora.std11g2.db)' ACL='owner:oracle:rwx,pgrp:oinstall:rwx,other::r--' PLACEMENT='favored' HOSTING_MEMBERS='rhel6m1'"




Check cluster resource failing over and dbvctl daemon starting when database instance goes down. The appsvip1 is the VIP used for dbvisit standby in the RAC configuration and dbvagentd and dbvnetd are cluster resource created for dbvagent and dbvnet.
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
appsvip1                            app.appvip.type           ONLINE             ONLINE             rhel6m1
dbvagentd                           cluster_resource          ONLINE             ONLINE             rhel6m1
dbvctld                             cluster_resource          ONLINE             ONLINE             rhel6m1
dbvnetd                             cluster_resource          ONLINE             ONLINE             rhel6m1
...
ora.std11g2.db                      ora.database.type         ONLINE             ONLINE             rhel6m1
ora.std11g2.db                      ora.database.type         ONLINE             ONLINE             rhel6m1
Stop the instance on node where dbvctl daemon is currently running. As there's a hard dependency between DB instance and dbvctl daemon -f option must be used to stop the DB instance.
srvctl stop instance -d std11g2 -i std11g21 -f
Check the cluster resource status.
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
appsvip1                            app.appvip.type           ONLINE             ONLINE             rhel6m2
dbvagentd                           cluster_resource          ONLINE             ONLINE             rhel6m2
dbvctld                             cluster_resource          ONLINE             ONLINE             rhel6m2
dbvnetd                             cluster_resource          ONLINE             ONLINE             rhel6m2
...
ora.std11g2.db                      ora.database.type         OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type         ONLINE             ONLINE             rhel6m2

Saturday, December 7, 2019

Enabling Parallel DML

In pre-12c versions to parallel DML was enabled via alter session. 12c introduced new hint to enable parallel DML via a SQL statement hint /*+ ENABLE_PARALLEL_DML */. As per the documentation "when parallel DML is enabled in a SQL statement with the ENABLE_PARALLEL_DML hint, only that specific statement is considered for parallel execution. However, even if parallel DML is enabled, the DML operation may still execute serially if there are no parallel hints or no tables with a parallel attribute or if restrictions on parallel operations are violated".
The post list a simple example of updating a table.

Updating without any hints
No surprise here, a simple full table scan happens.
SQL> update bigtable set OBJECT_TYPE='abc';

Execution Plan
----------------------------------------------------------
Plan hash value: 898092282

-------------------------------------------------------------------------------
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT   |          |  9854 | 78832 |    50   (2)| 00:00:01 |
|   1 |  UPDATE            | BIGTABLE |       |       |            |          |
|   2 |   TABLE ACCESS FULL| BIGTABLE |  9854 | 78832 |    50   (2)| 00:00:01 |
-------------------------------------------------------------------------------
Updating with parallel hint
Only the query portion gets parallelized.
SQL> update /*+ parallel */  bigtable set OBJECT_TYPE='abc';

Execution Plan
----------------------------------------------------------
Plan hash value: 1905498248

---------------------------------------------------------------------------------------------------------------
| Id  | Operation             | Name     | Rows  | Bytes | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT      |          |  9854 | 78832 |    27   (0)| 00:00:01 |        |      |            |
|   1 |  UPDATE               | BIGTABLE |       |       |            |          |        |      |            |
|   2 |   PX COORDINATOR      |          |       |       |            |          |        |      |            |
|   3 |    PX SEND QC (RANDOM)| :TQ10000 |  9854 | 78832 |    27   (0)| 00:00:01 |  Q1,00 | P->S | QC (RAND)  |
|   4 |     PX BLOCK ITERATOR |          |  9854 | 78832 |    27   (0)| 00:00:01 |  Q1,00 | PCWC |            |
|   5 |      TABLE ACCESS FULL| BIGTABLE |  9854 | 78832 |    27   (0)| 00:00:01 |  Q1,00 | PCWP |            |
---------------------------------------------------------------------------------------------------------------

Note
-----
   - automatic DOP: Computed Degree of Parallelism is 2
   - PDML is disabled in current session
The notes indicate Auto DOP of 2 was used but parallel DML (PDML) was not used.

Updating with enable parallel dml hint
SQL> update /*+ ENABLE_PARALLEL_DML */  bigtable set OBJECT_TYPE='abc';

Execution Plan
----------------------------------------------------------
Plan hash value: 898092282

-------------------------------------------------------------------------------
| Id  | Operation          | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT   |          |  9854 | 78832 |    50   (2)| 00:00:01 |
|   1 |  UPDATE            | BIGTABLE |       |       |            |          |
|   2 |   TABLE ACCESS FULL| BIGTABLE |  9854 | 78832 |    50   (2)| 00:00:01 |
-------------------------------------------------------------------------------

Note
-----
   - PDML disabled because object is not decorated with parallel clause
Nothing is parallelized. Reason is mentioned in the notes "object is not decorated with parallel clause".



Updating with enable parallel dml and parallel hint
SQL>  update /*+ ENABLE_PARALLEL_DML parallel */  bigtable set OBJECT_TYPE='abc';

Execution Plan
----------------------------------------------------------
Plan hash value: 1693571574

---------------------------------------------------------------------------------------------------------------
| Id  | Operation             | Name     | Rows  | Bytes | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT      |          |  9854 | 78832 |    27   (0)| 00:00:01 |        |      |            |
|   1 |  PX COORDINATOR       |          |       |       |            |          |        |      |            |
|   2 |   PX SEND QC (RANDOM) | :TQ10000 |  9854 | 78832 |    27   (0)| 00:00:01 |  Q1,00 | P->S | QC (RAND)  |
|   3 |    UPDATE             | BIGTABLE |       |       |            |          |  Q1,00 | PCWP |            |
|   4 |     PX BLOCK ITERATOR |          |  9854 | 78832 |    27   (0)| 00:00:01 |  Q1,00 | PCWC |            |
|   5 |      TABLE ACCESS FULL| BIGTABLE |  9854 | 78832 |    27   (0)| 00:00:01 |  Q1,00 | PCWP |            |
---------------------------------------------------------------------------------------------------------------

Note
-----
   - automatic DOP: Computed Degree of Parallelism is 2
Update statement now sits under the parallel coordinator. Notes indicate degree of parallelism used.
The degree of parallelism could be explicitly stated in the parallel clause along with the enable parallel dml.
SQL> update /*+ ENABLE_PARALLEL_DML parallel(4) */ bigtable set OBJECT_TYPE='abc';

Execution Plan
----------------------------------------------------------
Plan hash value: 1693571574

---------------------------------------------------------------------------------------------------------------
| Id  | Operation             | Name     | Rows  | Bytes | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT      |          |  9854 | 78832 |    14   (0)| 00:00:01 |        |      |            |
|   1 |  PX COORDINATOR       |          |       |       |            |          |        |      |            |
|   2 |   PX SEND QC (RANDOM) | :TQ10000 |  9854 | 78832 |    14   (0)| 00:00:01 |  Q1,00 | P->S | QC (RAND)  |
|   3 |    UPDATE             | BIGTABLE |       |       |            |          |  Q1,00 | PCWP |            |
|   4 |     PX BLOCK ITERATOR |          |  9854 | 78832 |    14   (0)| 00:00:01 |  Q1,00 | PCWC |            |
|   5 |      TABLE ACCESS FULL| BIGTABLE |  9854 | 78832 |    14   (0)| 00:00:01 |  Q1,00 | PCWP |            |
---------------------------------------------------------------------------------------------------------------

Note
-----
   - Degree of Parallelism is 4 because of hint

Thursday, December 5, 2019

CLSRSC-180: An error occurred while executing the command 'asmcmd afd_dsset'

Execution of roothas.pl failed during the reconfiguring of a 19.5 Oracle restart setup after a hostname change
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-7-244/crsconfig/roothas_2019-12-03_02-23-25PM.log
2019/12/03 14:23:31 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-7-244 successfully pinned.
2019/12/03 14:24:43 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

ip-172-31-7-244     2019/12/03 14:26:31     /opt/app/oracle/crsdata/ip-172-31-7-244/olr/backup_20191203_142631.olr     3329448500
2019/12/03 14:26:44 CLSRSC-180: An error occurred while executing the command '/opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset ""'
Died at /opt/app/oracle/product/19.x.0/grid/crs/install/oraafd.pm line 1943.


The alert log had following entries related to the issue at hand.
2019-12-03 14:39:58: Executing the step [] to postConfig AFD on the SIHA node
2019-12-03 14:39:58: AFD Diskstring:
2019-12-03 14:39:58: Running as user oracle: /opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset ""
2019-12-03 14:40:11: Removing file /tmp/IumY69Y5LW
2019-12-03 14:40:11: Successfully removed file: /tmp/IumY69Y5LW
2019-12-03 14:40:11: pipe exit code: 65280
2019-12-03 14:40:11: /bin/su exited with rc=255

2019-12-03 14:40:11: /opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset "" has failed to run with status 255
2019-12-03 14:40:11: Executing cmd: /opt/app/oracle/product/19.x.0/grid/bin/clsecho -p has -f clsrsc -m 180 '/opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset ""'
2019-12-03 14:40:11: Executing cmd: /opt/app/oracle/product/19.x.0/grid/bin/clsecho -p has -f clsrsc -m 180 '/opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset ""'
2019-12-03 14:40:11: Command output:
>  CLSRSC-180: An error occurred while executing the command '/opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset ""'
>End Command output
2019-12-03 14:40:11: CLSRSC-180: An error occurred while executing the command '/opt/app/oracle/product/19.x.0/grid/bin/asmcmd afd_dsset ""'
2019-12-03 14:40:11: ###### Begin DIE Stack Trace ######
2019-12-03 14:40:11:     Package         File                 Line Calling
2019-12-03 14:40:11:     --------------- -------------------- ---- ----------
2019-12-03 14:40:11:  1: main            roothas.pl            155 crsutils::dietrap
2019-12-03 14:40:11:  2: oraClusterwareComp::oraafd oraafd.pm            1943 main::__ANON__
2019-12-03 14:40:11:  3: oraClusterwareComp::oraafd oraafd.pm             287 oraClusterwareComp::oraafd::postConfigAFDSteps
2019-12-03 14:40:11:  4: oraClusterwareComp oraClusterwareComp.pm  123 oraClusterwareComp::oraafd::postConfigureSIHA
2019-12-03 14:40:11:  5: crsinstall      crsinstall.pm        1715 oraClusterwareComp::postConfigureCurrentNode
2019-12-03 14:40:11:  6: crsinstall      crsinstall.pm        1403 crsinstall::siha_post_config
2019-12-03 14:40:11:  7: crsinstall      crsinstall.pm         442 crsinstall::HAInstall
2019-12-03 14:40:11:  8: main            roothas.pl            264 crsinstall::new
2019-12-03 14:40:11: ####### End DIE Stack Trace #######
Solution specified in 2375603.1 didn't resolve it. But as per 29013832.8 the issue is related to bug 29013832 affecting 19.1 and 18.1. There's a patch (29013832 ) available to be applied on top of the 19.5.0.0.191015DBRU (pre-req 30125133).
From the failed state shown above run the patch. This will configure the OHAS and any future deconfig/config runs will not encounter the error.