Wednesday, February 4, 2015

Upgrading RAC from 12.1.0.1 to 12.1.0.2 - Database

After upgrading the grid infrastructure from 12.1.0.1 to 12.1.0.2 the next step is the upgrade of RAC software and the database itself. There are earlier post which upgraded 12.1.0.1 to 12.1.0.2 in data guard configuration and with PDBs. These posts have useful metalink notes that could be benefitial in RAC environment as well.
The database software upgrade is an out of place upgrade with role separation. Use orachk or cluvfy to check the pre upgrade status. Output from cluvfy is shown below.
cluvfy stage  -pre dbinst -upgrade -src_dbhome /opt/app/oracle/product/12.1.0/dbhome_1 -dest_dbhome /opt/app/oracle/product/12.1.0/dbhome_2 -dest_version 12.1.0.2.0

Performing pre-checks for database installation

Checking node reachability...
Node reachability check passed from node "rhel12c1"


Checking user equivalence...
User equivalence check passed for user "oracle"
Specify user name for database "rac12c1" [default "DBSNMP"] : system
Specify password for user "system"  in database "rac12c1" :

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c2,rhel12c1
TCP connectivity check passed for subnet "192.168.1.0"


Check: Node connectivity using interfaces on subnet "192.168.0.0"
Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c2,rhel12c1
TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rhel12c2:/opt/app/oracle/product/12.1.0/dbhome_2,rhel12c2:/tmp"
Free disk space check passed for "rhel12c1:/opt/app/oracle/product/12.1.0/dbhome_2,rhel12c1:/tmp"
Check for multiple users with UID value 500 passed
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Group existence check passed for "asmdba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Membership check for user "oracle" in group "asmdba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
There are no oracle patches required for home "/opt/app/oracle/product/12.1.0/dbhome_1".
There are no oracle patches required for home "/opt/app/oracle/product/12.1.0/dbhome_2".
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed
Default user file creation mask check passed

Checking CRS integrity...

Clusterware version consistency passed.

CRS integrity check passed

Checking Cluster manager integrity...


Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


Checking node application existence...

Checking existence of VIP node application (required)
VIP node application check passed

Checking existence of NETWORK node application (required)
NETWORK node application check passed

Checking existence of ONS node application (optional)
ONS node application check passed

Oracle Clusterware is installed on all nodes.
CTSS resource check passed
Query of CTSS for time offset passed

CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed
Checking integrity of file "/etc/resolv.conf" across nodes

"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rhel12c1,rhel12c2
checking DNS response from all servers in "/etc/resolv.conf"

Check for integrity of file "/etc/resolv.conf" failed

Time zone consistency check passed

Checking Single Client Access Name (SCAN)...

Checking TCP connectivity to SCAN listeners...
TCP connectivity to SCAN listeners exists on all cluster nodes

Checking name resolution setup for "rhel12c-scan.rac.domain.net"...

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking SCAN IP addresses...
Check of SCAN IP addresses passed

Verification of SCAN VIP and listener setup passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking stale database schema statistics...
Database stale schema statistics check is passed


ASM and CRS versions are compatible
Database Clusterware version compatibility passed.
OS user consistency check for upgrade successful
Group existence check passed for "asmadmin"
Membership check for user "grid" in group "asmadmin" passed
Group existence check passed for "asmoper"
Membership check for user "grid" in group "asmoper" passed
Group existence check passed for "asmdba"
Membership check for user "grid" in group "asmdba" passed

Pre-check for database installation was unsuccessful on all the nodes.
The failure is due to PRVF-5636 which is ignored in this case.



Once the pre-req checks are validated run the installer from the 12.1.0.2 installation media. Give a new home for Oracle database software (part of out of place upgrade).
Upgrade summary
Once the database software is installed next step is upgrade of the database. The database in this case is a non-CDB database and will be upgraded using DBUA. As this is a role separated environment check the ownership of the oracle binaries are correct set to (oracle:asmadmin). If not set the correct permission using setasmgidwrap.
Refer earlier posts (non-CDB upgrade,CDB+PDB upgrade) for additional information with regard to pre-reqs for upgrades using DBUA.
After running DBUA from the new 12.1.0.2 oracle home select upgrade option.
Select the database to be upgraded. DBUA detects the database type as RAC.
DBUA summary
Upgrade results
Verify the post upgrade status using orachk (orachk -u -o post). When satisfied with the upgrade status upgrade the compatible parameter. In RAC this parameter is expected to be same on all instances and cannot be updated in a rolling fashion (1636681.1).
At the end of the database upgrade the release version will also reflect the new version
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
Apply the patch 19518079 so that database upgrade status is reflected on registry$history.
This conclude the upgrade of RAC database from 12.1.0.1 to 12.1.0.2.

Related Posts
Upgrading 12c CDB and PDB from 12.1.0.1 to 12.1.0.2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure
Upgrade Oracle Database 12c1 from 12.1.0.1 to 12.1.0.2
Upgrading from 10.2.0.4 to 10.2.0.5 (Clusterware, RAC, ASM)
Upgrade from 10.2.0.5 to 11.2.0.3 (Clusterware, RAC, ASM)
Upgrade from 11.1.0.7 to 11.2.0.3 (Clusterware, ASM & RAC)
Upgrade from 11.1.0.7 to 11.2.0.4 (Clusterware, ASM & RAC)
Upgrading from 11.1.0.7 to 11.2.0.3 with Transient Logical Standby
Upgrading from 11.2.0.1 to 11.2.0.3 with in-place upgrade for RAC
In-place upgrade from 11.2.0.2 to 11.2.0.3
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 1
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure

Useful metalink notes
Master Note For Oracle Database 12c Release 1 (12.1) Database/Client Installation/Upgrade/Migration Standalone Environment (Non-RAC) [ID 1520299.1]
Complete Checklist for DBUA Upgrade from 12.1.0.1 to 12.1.0.N [ID 1933011.1]
How to Download and Run Oracle's Database Pre-Upgrade Utility [ID 884522.1]
How to execute sql scripts in Multitenant environment (catcon.pl) [ID 1932340.1]
Complete checklist for 12c R1 PDB upgrade (Upgrading single/multiple PDB) [ID 1933391.1]
Complete checklist for manual upgrade from 12.1.0.1 to 12.1.0.N (Full CDB Upgrade) [ID 1932762.1]
Oracle Database 12c Release 1 (12.1) DBUA : Understanding New Changes with All New 12.1 DBUA [ID 1493645.1]
Complete Checklist for Upgrading to Oracle Database 12c Release 1 using DBUA [ID 1516557.1]
Updating the RDBMS DST version in 12c Release 1 (12.1.0.1 and up) using DBMS_DST [ID 1509653.1]
dba_registry_history Table On Newly Created 11.2.0.2 Database Shows PSU Entry[ID 1367065.1]
"VIEW INVALIDATE" In Action Column Dba_Registry_History During Upgarde To 11.2.0.2 [ID 1390402.1]