Monday, August 23, 2010

CRS-2640 when ASM diskgroup dependencies not automatically updated on 11.2

Datbase is a single instance 11gR2 RAC. It already has two diskgroups configured namely DATA and FLASH. A third diskgroup is added to the ASM instance and a tablespace is created using this group.
After sometime the tablespace and diskgroup is dropped but START_DEPENDENCIES and STOP_DEPENDENCIES added to the database resource is still persistent and prevents the restart of the database.

1. Create the diskgroup
sqlplus / as sysasm
create diskgroup reduntest normal redundancy failgroup fail1 disk 'ORCL:RED1' failgroup fail2 disk 'ORCL:RED2'
failgroup fail3 disk 'ORCL:RED3' failgroup fail4 disk 'ORCL:RED4' attribute 'compatible.asm'='11.1';

Diskgroup created.
2. Create the tablespace
sqlplus / as sysdba
SQL> create tablespace reduntbs datafile '+REDUNTEST(datafile)' SIZE 1g;

Tablespace created.
3. Verify the dependencies on the database with crsctl stat res -p. Here grep has been used to list the above mention diskgroup dependency only
crsctl stat res -p | grep REDUNTEST
NAME=ora.REDUNTEST.dg
START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg,ora.REDUNTEST.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg,ora.FLASH.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FLASH.dg,shutdown:ora.REDUNTEST.dg)
4. Drop the tablespace and check the dependcies, and diskgroup is still listed as a dependency
SQL> drop tablespace reduntbs including contents and datafiles cascade constraints;

Tablespace dropped.

crsctl stat res -p | grep REDUNTEST
NAME=ora.REDUNTEST.dg
START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg,ora.REDUNTEST.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg,ora.FLASH.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FLASH.dg,shutdown:ora.REDUNTEST.dg)
5. But this is not listed on srvctl config
srvctl config database -d clusdb -a
Database unique name: clusdb
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/clusdb
Oracle user: oracle
Spfile: +DATA/clusdb/spfileclusdb.ora
Domain: domain.net
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: clusdb
Database instances: clusdb1
Disk Groups: DATA,FLASH
Services:
Database is enabled
Database is administrator managed
6. Even after dropping the diskgroup the dependency is still there
SQL> drop diskgroup reduntest;

Diskgroup dropped.

crsctl stat res -p | grep REDUNTEST
START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg,ora.REDUNTEST.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg,ora.FLASH.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FLASH.dg,shutdown:ora.REDUNTEST.dg)
7. Shutdown of the database works fine but start throws a dependency error
srvctl stop database -d clusdb
srvctl start database -d clusdb
PRCR-1079 : Failed to start resource ora.clusdb.db
CRS-2640: Required resource 'ora.REDUNTEST.dg' is missing.
8. Solution is to explicity set the diskgroups using srvctl, which removes the dependencies
srvctl modify database -d clusdb -a 'DATA,FLASH'

crsctl stat res -p | grep REDUNTEST
and doesn't show anything.