Recover From a Clusterware Home Deletion

This blog is as a result of the outcome seen twice on node deletion step 14.
It is assumed that all the clusterware files inside the CRS_HOME has been deleted and at best only the directories are available without any files in them.

In this case a new clusterare installation is needed unless those clusterware homes (each home individually) are backed up using some OS utility (this option hasn't been tested).

Even though clusterware is newly installed oracle home and asm home is intact and are cluster aware. So what is required is to add the existing homes, database, instances and etc to the new clusterware and ocr to be managed by the cluster.

1. Remove startup commands from /etc/inittab

2. Remove symbolic links and init scripts from init.d and rc#.d

cd /etc/rc.d
for i in `find . -name S96*`; do rm -f $i; done
for i in `find . -name K19*`; do rm -f $i; done

3. Remove directory with ocr location and scr settings.

rm -rf /etc/oracle

4. Change permission on clusterware files (ocr, vote) location

chown oracle:oinstall /dev/raw/raw1
chown oracle:oinstall /dev/raw/raw2

5. Zero out the block or character devices used for ocr and vote disk storage

dd if=/dev/zero of=/dev/raw/raw1 bs=8192 count=1024
dd if=/dev/zero of=/dev/raw/raw2 bs=8192 count=1024

6. Backup the current inventory folder and create a new empty inventory folder (oraInventory) change the permission on it to 770 and oracle:oinstall

7. Remove the crs home and create a new directory with ownership oracle:oinstall. Execute runInstaller for the clusterware. If clusterware has been upgraded, started with 10.2 and was upgraded to 11.1 when files got deleted, then start from the initial version if ocr and vote disk location doesn't have enough space to accomodate the new version and upgrade afterwards to match the version before the files got deleted. Once the installation is complete attache the other oracle homes (ORACLE_HOME, ASM_HOME) to new inventory.

./runInstaller -silent -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME="oracle_home" "CLUSTER_NODES={rac1,rac2}"

Once the installation is completed add listener, asm, database and instances.

8. Add listener

srvctl add listener -n rac1 -o $ORACLE_HOME   
srvctl add listener -n rac2 -o $ORACLE_HOME
srvctl start listener -n rac1
srvctl start listener -n rac2

9. Add asm

srvctl add asm -n rac1 -i +ASM1 -o $ORACLE_HOME
srvctl add asm -n rac2 -i +ASM2 -o $ORACLE_HOME
srvctl start asm -n rac1
srvctl start asm -n rac2

10. Add database

srvctl add database -d racdb -o $ORACLE_HOME -p '+DATA/racdb/spfileracdb.ora'
srvctl modify database -d racdb -o $ORACLE_HOME -p '+DATA/racdb/spfileracdb.ora' -m domain.net

11. Add instance

srvctl add instance -d racdb -i racdb1 -n rac1
srvctl modify instance -d racdb -i racdb1  -s +ASM1

srvctl add instance -d racdb -i racdb2 -n rac2
srvctl modify instance -d racdb -i racdb2 -s +ASM2

12. Start the database

srvctl start database -d racdb