Wednesday, April 1, 2015

Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2

The system used for upgrade is the same system featured on an earlier post (upgrading single instance with ASM from 11.2.0.3 to 11.2.0.4). This post list the main points to look out for when upgrading the grid infrastructure to 12.1.0.2. Only the grid infrastructure is upgraded as the database software is on standard edition (SE) and at the time of this post there's no SE for 12.1.0.2.
1. If the new GI installer is unable to detect the existing GI then it could be same issue mentioned in the previous post. Fix it by updating the inventory information with crs=true. Also check if $ORACLE_BASE/cfgtoollogs has write permission for grid user.
Also prior to upgrade make sure ASM SPfile is residing in ASM not on local file system (@/dbs/spfile+ASM.ora). ASM upgrade would fail with
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid2/crs/install/crsconfig_params

Upgrading ASM failed with the following message:
ORA-32002: cannot create SPFILE already being used by the instance
or
Upgrading ASM failed with the following message:
ORA-01565: error in identifying file '/opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora'
ORA-27037: unable to obtain file status
if ASM SPfile is on the local file system.

2. Run the 12c GI installer and select upgrade GI and ASM option. New GI install is done as an out of place upgrade.

3. When prompted the rootupgrade.sh script.
 /opt/app/oracle/product/12.1.0/grid_1/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/oracle/product/12.1.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid_1/crs/install/crsconfig_params

ASM Configuration upgraded successfully.

Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel6m1 successfully pinned.
2015/03/12 15:30:10 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

2015/03/12 15:31:14 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

2015/03/12 15:33:01 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p first'

2015/03/12 15:33:11 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p last'


rhel6m1     2015/03/12 15:33:12     /opt/app/oracle/product/12.1.0/grid_1/cdata/rhel6m1/backup_20150312_153312.olr     0

rhel6m1     2015/03/12 12:41:21     /opt/app/oracle/product/11.2.0/grid_4/cdata/rhel6m1/backup_20150312_124121.olr     -

rhel6m1     2015/03/11 17:57:18     /opt/app/oracle/product/11.2.0/grid_1/cdata/rhel6m1/backup_20150311_175718.olr     -
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2015/03/12 15:38:33 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
4. Since the database remains in 11.2.0.4 version only the compatible.asm attribute is updated to 12c version.
alter diskgroup data set attribute 'compatible.asm'='12.1.0.2.0';
alter diskgroup flash set attribute 'compatible.asm'='12.1.0.2.0';
5. This concludes the upgrade of GI in a single instance environment with ASM. The software version will have the 12c version while release version remaining on 11.2.0.4
$  crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.1.0.2.0]


One thing that was noticed after the upgrade is that trying to access ASM config did not return any results but an error.
$ srvctl config asm
ASM home: 
PRCA-1057 : Failed to retrieve the password file location used by ASM asm
PRCR-1097 : Resource attribute not found: PWFILE
Trying to modify the pw file location didn't work either
$ srvctl modify asm -pwfile /opt/app/oracle/product/12.1.0/grid_1/dbs/orapw+ASM
PRCR-1097 : Resource attribute not found: CARDINALITY
Only way to fix this is to stop all the resources and recreate the ASM config.
crsctl stop resource -all
srvctl remove asm -force
srvctl add asm -listener listener -spfile "+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.874087179" -pwfile /opt/app/oracle/product/12.1.0/grid_1/dbs/orapw+ASM -diskstring "/dev/sd*"

crsctl start resource -all

srvctl config asm
ASM home: 
Password file: /opt/app/oracle/product/12.1.0/grid_1/dbs/orapw+ASM
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.874087179
ASM diskgroup discovery string: /dev/sd*
This is a known issue on 12c. For more follow MOS notes.

Useful metalink notes
Oracle Restart: WARNING: unknown state for ASM password file location resource, Return Value: 3 [ID 1935891.1]
Alternative Way To Upgrade An ASM Standalone Configuration From Release 11.2.0.<#> to release 12.1.0.<#>. [ID 1964405.1]
Reconfiguring & Recreating The 11gR2/12cR1 Restart/OHAS/SIHA Stack Configuration (Standalone). [ID 1422517.1]
Things to Consider Before Upgrading to 12.1.0.2 to Avoid Poor Performance or Wrong Results [ID 2034610.1]

Related Posts
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading RAC from 12.1.0.1 to 12.1.0.2 - Grid Infrastructure
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Grid Infrastructure
Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Database