Tuesday, July 9, 2013

Upgrading from 11.2.0.3 to 12.1.0.1 RAC

The earlier post showed the steps for upgrading grid infrastructure from 11gR2 to 12c. This post list the steps for upgrading the RAC software to 12c. Since 11gR2 upgrades are out of place as such upgrading the database software requires no downtime and database is not involved in the upgrade process until the actual database upgrade happens with DBUA (or manually).
cluvfy could be used to verify the upgrade readiness. This will be run as Oracle user (as opposed to grid user in the earlier post).
[oracle@rhel6m1 database]$  /opt/app/12.1.0/grid/bin/cluvfy stage -pre dbinst -upgrade -src_dbhome /opt/app/oracle/product/11.2.0/dbhome_1 -dbname std11g2 -dest_dbhome /opt/app/oracle/product/12.1.0/dbhome_1 -dest_version 12.1.0.1.0 -fixup

Performing pre-checks for database installation

Checking node reachability...
Node reachability check passed from node "rhel6m1"


Checking user equivalence...
User equivalence check passed for user "oracle"
Specify user name for database "std11g2" [default "DBSNMP"] : system
Specify password for user "system"  in database "std11g2" :

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.0.0"
Node connectivity passed for subnet "192.168.0.0" with node(s) rhel6m2,rhel6m1
TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) rhel6m2,rhel6m1
TCP connectivity check passed for subnet "192.168.1.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rhel6m2:/opt/app/oracle/product/12.1.0/dbhome_1,rhel6m2:/tmp"
Free disk space check passed for "rhel6m1:/opt/app/oracle/product/12.1.0/dbhome_1,rhel6m1:/tmp"
Check for multiple users with UID value 501 passed
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Group existence check passed for "asmdba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Membership check for user "oracle" in group "asmdba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
There are no oracle patches required for home "/opt/app/oracle/product/11.2.0/dbhome_1".
There are no oracle patches required for home "/opt/app/oracle/product/12.1.0/dbhome_1".
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed
Default user file creation mask check passed

Checking CRS integrity...

Clusterware version consistency passed.

CRS integrity check passed

Checking Cluster manager integrity...


Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


Checking node application existence...

Checking existence of VIP node application (required)
VIP node application check passed

Checking existence of NETWORK node application (required)
NETWORK node application check passed

Checking existence of ONS node application (optional)
ONS node application check passed


Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
CTSS resource check passed


Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed
Checking integrity of file "/etc/resolv.conf" across nodes

"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" passed

Time zone consistency check passed

Checking Single Client Access Name (SCAN)...

Checking TCP connectivity to SCAN Listeners...
TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "rhel6m-scan"...

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking SCAN IP addresses...
Check of SCAN IP addresses passed

Verification of SCAN VIP and Listener setup passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.
Checking stale database schema statistics...
PRVG-11143 : The following error occurred during stale database schema statistics check.
PRVG-11115 : Following error occurred while establishing connection to database "std11g2"
PRCQ-1000 : An error occurred while establishing connection to database with user name "system" and connect descriptor:
(DESCRIPTION = (LOAD_BALANCE=on)  (ADDRESS = (PROTOCOL = TCP)(HOST = rhel6m-scan)(PORT = null)) (CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = std11g2)))
IO Error: The Network Adapter could not establish the connection


ASM and CRS versions are compatible
Database Clusterware version compatibility passed.
OS user consistency check for upgrade successful


NOTE:
No fixable verification failures to fix

Pre-check for database installation was unsuccessful.
Checks did not pass for the following node(s):
        std11g2
The cluvfy ask for database username to check stale database schema statistics but the connection description has null value for port. This is similar to what was seen after changing the listener and scan listener names. So this was ignored and all other pre-reqs were successful.
Oracle has introduced several OS user group for further job role separation (for backup, data guard broker and key management). If these to be used then these groups could be created before the 12c software is installed.
[root@rhel6m1 ~]# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),506(asmdba)
[root@rhel6m1 ~]# groupadd backupdba
[root@rhel6m1 ~]# groupadd dgdba
[root@rhel6m1 ~]# groupadd kmdba
[root@rhel6m1 ~]# usermod -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba oracle
[root@rhel6m1 ~]# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),506(asmdba),508(backupdba),509(dgdba),510(kmdba)
As 12c RAC upgrade is an out of place upgrade there's no down time on the database.




Execute runInstaller to start the installation/upgrade of RAC.

Once the software is installed copy the COST related files (sqlnet.ora and if desired tnsnames.ora) to new Oracle Home so when the database is upgraded from the new home it is able to register with the listeners. (read update on 31/07/2013)
This concludes the 11gR2 to 12c RAC software upgrade, the remaining step is to upgrade the database.

Related Post
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure
Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Database