Wednesday, November 15, 2017

Unable to obtain current patch information due to error: 20001, ORA-20001: Latest xml inventory is not loaded into table

After upgrading single instance oracle restart setup to 12.2 from 11.2.0.4 following is observed in the alert log
Unable to obtain current patch information due to error: 20001, ORA-20001: Latest xml inventory is not loaded into table
ORA-06512: at "SYS.DBMS_QOPATCH", line 777
ORA-06512: at "SYS.DBMS_QOPATCH", line 864
ORA-06512: at "SYS.DBMS_QOPATCH", line 2222
ORA-06512: at "SYS.DBMS_QOPATCH", line 740
ORA-06512: at "SYS.DBMS_QOPATCH", line 2247
Checking in the $ORACLE_HOME/QOpatch/qopatch_log.log showed
KUP-05007:   Warning: Intra source concurrency disabled because the preprocessor option is being used.

Field Definitions for table OPATCH_XML_INV
  Record format DELIMITED BY NEWLINE
  Data in file has same endianness as the platform
  Reject rows with all null fields

  Fields in Data Source:

    XML_INVENTORY                   CHAR (100000000)
      Terminated by "UIJSVTBOEIZBEFFQBL"
      Trim whitespace same as SQL Loader
KUP-04095: preprocessor command /opt/app/oracle/product/12.2.0/dbhome_1/QOpatch/qopiprep.bat encountered error "/opt/app/oracle/product/12.2.0/dbhome_1/QOpatch/qopiprep.bat: 
line 55: /opt/app/oracle/product/12.2.0/dbhome_1/rdbms/log/stout_std11g2.txt: Permission denied
As per 1602089.1 this is due to running the command with a different user who doesn't have permission on $ORACLE_HOME/QOpatch folder. But in this case it was not ture, the DB start was done by oracle user who has permissions on the QOPatch folder. Even then the same error was shown on alert log.



This is a role separate single instance setup and GI is owned by grid user. As mentioned in the password store post, in single instance setup with role separation has the problem of certain commands being run as grid user instead of oracle user. In this case the grid user doesn't have write permission on $ORACLE_HOME/rdbms/log folder which has 755 as the permission. That's the root cause of the issue. Change this log directory permission to 775 so that grid user is able to write to it. After this during the start up the full patch list is shown on the alert log.
2017-10-20T14:46:26.105603+01:00
===========================================================
Dumping current patch information
===========================================================
Patch Id: 26710464
Patch Description: Database Release Update : 12.2.0.1.171017 (26710464)
Patch Apply Time: 2017-10-20T13:05:21+01:00
Bugs Fixed: 14690846,17027695,17533661,19285025,19327292,19614243,20003668,
20324049,20620169,20736227,21159907,21186167,21981529,21985256,22072543,
22087683,22179537,22446455,22503297,22568728,22594071,22628825,22645009,
22654475,22729345,22898198,22950945,22981722,23026585,23035249,23055900,
23061453,23125560,23151677,23179662,23234232,23300142,23481673,23491861,
23499160,23521523,23527363,23548817,23581777,23599216,23665623,23730961,
23733981,23735292,23746128,23749454,24289874,24326846,24332831,24334708,
24336249,24341675,24368004,24376875,24376878,24385983,24421668,24425998,
24457597,24485161,24485174,24509056,24534401,24555417,24556967,24560906,
24573817,24578718,24578797,24589590,24609996,24624166,24642495,24655717,
24664211,24668398,24674955,24676172,24677696,24693290,24714096,24717183,
24735430,24737064,24737403,24744686,24792678,24796092,24811725,24812047,
24827228,24831514,24835919,24850622,24907917,24908321,24912588,24923215,
24929210,24938784,24942749,24960044,24968162,24976007,25029022,25034396,
...
...
The issue was only observed in single instance role separated setup. In role separated RAC configuration there wasn't any permission issues.

Useful metalink notes
Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table [ID 1602089.1]

Wednesday, November 1, 2017

Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1

This post lists the steps for upgrading an Oracle restart environment (Single instance non-CDB on ASM) from 11.2.0.4 to 12.2.0.1. The 11.2.0.4 setup was on RHEL 6.4. The current versions of the GI are
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.4.0]
The Oracle restart environment used is a role separate setup where GI is installed as grid user while Oracle software is installed Oracle user. The GI and Oracle homes had the latest PSU applied (Database Patch Set Update : 11.2.0.4.171017 (26392168)). With this patch set applied orachk pre-upgrade did not list any patches to be applied before the upgrade. If the current patch set is different then additional patches may need to be applied before the upgrade. Refer 2180188.1 and earlier post of 11.2.0.4 to 12.2 RAC upgrade post for these patches.
Unlike the previous versions, 12.2 grid infrastructure requires unzipping it into a grid home. There's no GI home selection on the OUI. Create the new directory path for 12.2 GI and unzip the GI bundle.
mkdir -p /opt/app/oracle/product/12.2.0/grid
cp ~/linuxx64_12201_grid_home.zip /opt/app/oracle/product/12.2.0/grid
cd /opt/app/oracle/product/12.2.0/grid
unzip linuxx64_12201_grid_home.zip 
cluvfy from the 12.2 GI could be used to further check the system setup. This option is not for upgrade check but could be used to check readiness of oracle restart installation.
$ ./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/sbin,rhel6m1:/tmp ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.20.51.0.2 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libgcc-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...PASSED
Verifying Package: sysstat-9.0.4 ...PASSED
Verifying Package: gcc-4.4.7 ...PASSED
Verifying Package: gcc-c++-4.4.7 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.81 ...PASSED
Verifying Package: glibc-2.12 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.12 (x86_64) ...PASSED
Verifying Package: libaio-0.3.107 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.107 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-5.43-1 ...PASSED
Verifying Package: net-tools-1.60-110 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.
The failure encountered is due to under size swap which could be ignored.
ASM will be upgraded as part of the upgrade process. Therefore stop the database before starting the GI upgrade. Upgrade process will prompt to stop the database during gridSetup run as well.
srvctl stop database -d std11g2
Start the GI upgrade by running the gridSetup.sh from the grid home.
$ ./gridSetup.sh
This will start the GI installer and select the upgrade GI option.
OS groups for authentication into ASM.
Oracle base location could be selected but not the GI Home location. This will be fixed, based on where the GI software is extracted.
Summary page and progress.
When prompted execute the rootupgrade.sh
# /opt/app/oracle/product/12.2.0/grid/rootupgrade.sh
Performing root user operation.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/roothas_2017-10-19_05-13-24PM.log
2017/10/19 17:13:26 CLSRSC-363: User ignored prerequisites during installation

ASM has been upgraded and started successfully.

Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel6m1 successfully pinned.
2017/10/19 17:17:38 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
CRS-4123: Oracle High Availability Services has been started.

2017/10/19 17:19:36 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p first'
2017/10/19 17:20:01 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p last'

rhel6m1     2017/10/19 17:20:04     /opt/app/oracle/product/12.2.0/grid/cdata/rhel6m1/backup_20171019_172004.olr     0

rhel6m1     2015/03/16 14:14:51     /opt/app/oracle/product/11.2.0/grid_4/cdata/rhel6m1/backup_20150316_141451.olr     -

rhel6m1     2015/03/11 17:57:18     /opt/app/oracle/product/11.2.0/grid_1/cdata/rhel6m1/backup_20150311_175718.olr     -
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/10/19 17:21:35 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
The GI versions after the upgrade
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.2.0.1.0]
The next step of the upgrade is to install 12.2 database software. 12.2 introduced a new OS group for RAC DB management. If this is used then add the user group to both oracle and grid. Not adding the group to grid user result in an issue which is mentioned in 12.1.0.2 to 12.2.1.0 single instance upgrade post.
# groupadd racdba
# usermod -g oinstall -G dba,oper,asmoper,asmdba,asmadmin,backupdba,dgdba,kmdba,racdba oracle
# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmadmin),506(asmdba),507(asmoper),509(dgdba),510(kmdba),511(backupdba),512(racdba)
# usermod -g oinstall -G dba,asmoper,asmdba,asmadmin,racdba grid
id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),502(dba),505(asmadmin),506(asmdba),507(asmoper),512(racdba)
As GI home was install first some of the directories may not have write permission for oracle user. Change permission of those directories. In this case it was the 12.2.0 directory in the GI home path.
drwxr-xr-x. 3 grid oinstall 4096 Oct 19 16:44 12.2.0
$ chmod 775 12.2.0
drwxrwxr-x. 3 grid oinstall 4096 Oct 19 16:44 12.2.0
Install DB software selecting "software only" option.
RACDBA group selected for RAC administration.
Summary
Once the 12.2 DB software is installed next step is to upgrade the DB. 12.2 provides a preupgrade check jar file which need to be run using current oracle home (old version). Preupgrade.jar is available in 12.2_ORACLE_HOME/rdbms/admin
$ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STD11G2
     Container Name:  Not Applicable in Pre-12.1 database
       Container ID:  Not Applicable in Pre-12.1 database
            Version:  11.2.0.4.0
         Compatible:  11.2.0.4.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  14
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  R1

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Enterprise Manager Repository   [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             600 MB  DISABLED      551 MB  None
     SYSTEM                             700 MB  ENABLED       806 MB  None
     TEMP                                20 MB  ENABLED       150 MB  None
     UNDOTBS1                           225 MB  ENABLED       400 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.

   + (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

     The database contains 2 objects in the recycle bin.

     The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
   + Remove the EM repository.

     - Copy the $ORACLE_HOME/rdbms/admin/emremove.sql script from the target
     12.2.0.1.0 ORACLE_HOME into the source 11.2.0.4.0 ORACLE_HOME.

     Step 1: If database control is configured, stop EM Database Control,
     using the following command

       $> emctl stop dbconsole

     Step 2: Connect to the database using the SYS account AS SYSDBA

       SET ECHO ON;
       SET SERVEROUTPUT ON;
      @emremove.sql

     Without the set echo and serveroutput commands, you will not be able to
     follow the progress of the script.

     The database has an Enterprise Manager Database Control repository.

     Starting with Oracle Database 12c, the local Enterprise Manager Database
     Control does not exist anymore. The repository will be removed from your
     database during the upgrade.  This step can be manually performed before
     the upgrade to reduce downtime.

   + Run 11.2.0.4.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     1 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + Backup the existing ACLs and their assignments for reference. Use the
     new DBMS_NETWORK_ACL_ADMIN interfaces and dictionary views to administer
     network privileges after upgrade.

     The database contains network ACLs with privileges that will be migrated
     to a new format in 12c.

     Network access control list (ACL) privileges in 11g will be migrated to
     a new format in 12c. As part of the migration, new
     DBMS_NETWORK_ACL_ADMIN interfaces and dictionary views are provided, and
     privileges in the existing ACLs will be converted to the new format with
     new ACL names. The old ACL names, DBMS_NETWORK_ACL_ADMIN interfaces and
     dictionary views may continue to be used but are deprecated and their
     use is discouraged. For further information, refer to My Oracle Support
     note number 2078710.1.

   + Review and remove any unnecessary EVENTS.

     The database contains events.

     There are events set that should be removed before upgrade, unless your
     application vendors and/or Oracle Support state differently.  Changes
     will need to be made in the spfile.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 11.2.0.4
     Oracle Database Performance Tuning Guide.

   + Directly grant ADMINISTER DATABASE TRIGGER privilege to the owner of the
     trigger or drop and re-create the trigger with a user that was granted
     directly with such. You can list those triggers using "SELECT OWNER,
     TRIGGER_NAME FROM DBA_TRIGGERS WHERE BASE_OBJECT_TYPE=''DATABASE'' AND
     OWNER NOT IN (SELECT GRANTEE FROM DBA_SYS_PRIVS WHERE
     PRIVILEGE=''ADMINISTER DATABASE TRIGGER'')"

     There is one or more database triggers whose owner does not have the
     right privilege on the database.

     The creation of database triggers must be done by users granted with
     ADMINISTER DATABASE TRIGGER privilege. Privilege must have been granted
     directly.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 14 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.

   + Gather statistics on fixed objects two weeks after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

     This recommendation is given for all preupgrade runs.

     Fixed object statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans.  Those
     statistics are specific to the Oracle Database release that generates
     them, and can be stale upon database upgrade.

Preupgrade generated files:
    /opt/app/oracle/cfgtoollogs/std11g2/preupgrade/preupgrade_fixups.sql
    /opt/app/oracle/cfgtoollogs/std11g2/preupgrade/postupgrade_fixups.sql
Before the upgrade the DB component status and versions, plus the timezone file version.
 COMP_ID    COMP_NAME                                VERSION         STATUS
---------- ---------------------------------------- --------------- --------
XDB        Oracle XML Database                      11.2.0.4.0      VALID
CONTEXT    Oracle Text                              11.2.0.4.0      VALID
OWM        Oracle Workspace Manager                 11.2.0.4.0      VALID
CATALOG    Oracle Database Catalog Views            11.2.0.4.0      VALID
CATPROC    Oracle Database Packages and Types       11.2.0.4.0      VALID


FILENAME                VERSION
-------------------- ----------
timezlrg_14.dat              14
Read the output from preupgrade.jar and carry out necessary pre-upgrade tasks. Some are included in the preupgrade_fixups.sql while others may required to be carried out manually. Once pre-upgrade tasks are completed run dbua from the 12.2 home to begin the upgrade.
Select the DB for upgrade. In this case type is indicated as SI (single instance)
Select the upgrade option. The timezone is upgraded along with the DB upgrade.
Select the listener. In this case the listener port is a non-default port.
Upgrade summary and progress
Post upgrade summary
Run the post upgrade script
@postupgrade_fixups.sql
DB components versions and status after the upgrade
COMP_ID    COMP_NAME                                VERSION    STATUS
---------- ---------------------------------------- ---------- --------
CATALOG    Oracle Database Catalog Views            12.2.0.1.0 VALID
CATPROC    Oracle Database Packages and Types       12.2.0.1.0 VALID
OWM        Oracle Workspace Manager                 12.2.0.1.0 VALID
CONTEXT    Oracle Text                              12.2.0.1.0 VALID
XDB        Oracle XML Database                      12.2.0.1.0 VALID
Timezone file version after upgrade
FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_26.dat              26          0
As mentioned in 11.2.0.4 to 12.2.0.1 RAC upgrade post, during the upgrade new user accounts are added to DB with default passwords. Change the default password of these user accounts.

Check the registry history view for the upgrade entry.
ACTION_TIME                    ACTION               NAMESPACE   VERSION            ID COMMENTS
------------------------------ -------------------- ----------- ---------- ---------- ----------------------------------
11-MAR-15 06.48.08.733830 PM   APPLY                SERVER      11.2.0.3            0 Patchset 11.2.0.2.0
12-MAR-15 11.54.22.212281 AM   APPLY                SERVER      11.2.0.3           13 PSU 11.2.0.3.13
16-MAR-15 03.09.19.557408 PM   VIEW INVALIDATE                                8289601 view invalidation
16-MAR-15 03.09.19.634383 PM   UPGRADE              SERVER      11.2.0.4.0            Upgraded from 11.2.0.3.0
16-MAR-15 03.09.43.891843 PM   APPLY                SERVER      11.2.0.4            0 Patchset 11.2.0.2.0
10-AUG-15 04.28.56.297048 PM   APPLY                SERVER      11.2.0.4            7 PSU 11.2.0.4.7
19-OCT-17 04.22.55.222389 PM   APPLY                SERVER      11.2.0.4       171017 PSU 11.2.0.4.171017
                               BOOTSTRAP            DATAPATCH   12.2.0.1              RDBMS_12.2.0.1.0_LINUX.X64_170125
19-OCT-17 06.40.33.227204 PM   VIEW INVALIDATE                                8289601 view invalidation
19-OCT-17 06.42.41.868410 PM   UPGRADE              SERVER      12.2.0.1.0            Upgraded from 11.2.0.4.0
Upgrade the database and ASM compatible parameters
alter system set compatible='12.2.0.1.0' scope=spfile sid='*';

srvctl stop database -d std11g2

alter diskgroup flash SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup DATA  SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup flash set attribute 'compatible.rdbms'='12.2.0.1.0';
alter diskgroup data set attribute 'compatible.rdbms'='12.2.0.1.0';

GROUP_NUMBER NAME                 VALUE
------------ -------------------- ---------------
           1 compatible.asm       12.2.0.1.0
           1 compatible.rdbms     12.2.0.1.0
           2 compatible.asm       12.2.0.1.0
           2 compatible.rdbms     12.2.0.1.0

srvctl start database -d std11g2
Finally run orachk and cluvfy for check post upgrade state
./orachk -u -o post

cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.
Useful metalink notes
ACFS Drivers Install reports CLSRSC-400: A system reboot is required to continue installing [ID 2025056.1]
Patches to apply before upgrading Oracle GI and DB to 12.2.0.1 [ID 2180188.1]
12.2 Grid Infrastructure Installation: What's New [ID 2024946.1]
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]

Related Posts
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2

Wednesday, October 25, 2017

Downgrade Grid Infrastructure from 12.2.0.1 to 12.1.0.2

This post list steps for downgrading grid infrastructure from 12.2.0.1 to 12.1.0.2. The grid infrastructure upgrade has been successful on all nodes. If upgrade to 12.2 had failed on any of the nodes then follow oracle documentation for additional steps/options required for such cases. Only the GI was upgraded to 12.2 and the database remained on 12.1.0.2. The current version of the cluster and nodes is as below
$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel12c1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel12c2] is [12.2.0.1.0]
1. As the first step for downgrade, remove the MGMTDB.
[grid@rhel12c1 ~]$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.
2. Run rootcrs.sh with downgrade option from any of the nodes
[root@rhel12c1 grid]# ./crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel12c1/crsconfig/crsdowngrade_rhel12c1_2017-09-25_04-26-42PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c1'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/09/25 16:28:17 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:28:43 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:28:43 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:29:21 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:29:21 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
3. Run the downgrade from the last node.
[root@rhel12c2 grid]# ./crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel12c2/crsconfig/crsdowngrade_rhel12c2_2017-09-25_04-30-45PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSFS.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.FRA.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.GIMR.dg' is already running on 'rhel12c2'
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.std12c1.db' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2676: Start of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel12c2' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
2017/09/25 16:39:54 CLSRSC-338: Successfully downgraded OCR to version 12.1.0.2.0
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
2017/09/25 16:40:24 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:40:50 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:40:50 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:41:25 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:41:25 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2017/09/25 16:41:25 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/opt/app/12.1.0/grid2'.
2017/09/25 16:41:27 CLSRSC-592: Run 'crsctl start crs' from home /opt/app/12.1.0/grid2 on each node to complete downgrade.



4. Upgrade the oracle inventory by removing the CRS association from 12.2 GI home and adding it to 12.1 home.
[grid@rhel12c1 bin]$ pwd
/opt/app/12.2.0/grid/oui/bin

./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.2.0/grid "CLUSTER_NODES=rhel12c1,rhel12c2" -doNotUpdateNodeList 
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5119 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

[grid@rhel12c1 bin]$ pwd
/opt/app/12.1.0/grid2/oui/bin

./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES=rhel12c1,rhel12c2"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5119 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
Verify the update on inventory.xml
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel12c1"/>
      <NODE NAME="rhel12c2"/>
   </NODE_LIST>
</HOME>

<HOME NAME="OraGI12Home2" LOC="/opt/app/12.2.0/grid" TYPE="O" IDX="3"/>
5. If the /etc/oratab is missing any ASM references with downgraded 12.1 GI home, add the entries manually.

6. Before re-creatign the 12.1 MGMTDB check if there's any references to mgmtdb (using crsctl stat res -t). If there are any references then recreating the MGMTDB will fail with
PRCD-1000 : Database _mgmtdb already exists
PRCR-1086 : resource ora.mgmtdb is already registered
mgmtdb could be removed with
srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y
7. Once all residual information on mgmtdb is removed create the MGMTDB root container.
[grid@rhel12c1 ~]$ $GI_HOME/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName +GIMR -datafileJarLocation $GI_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
 Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
8. Create the PDB inside the MGMTDB root. The PDB should be named same as the cluster name. Cluster name could be found out with
$ cemutlo -n
prod-cluster
If cluster name has a dash (-) then when specify the PDB name replace the dash (-) with an underscore (_). Without this change the creation of the PDB will fail. More on this was mentioned on an earlier post.
[grid@rhel12c1 ~]$ $GI_HOME/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName prod_cluster -createPDBFrom RMANBACKUP -PDBBackUpfile  $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile  $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck

4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/prod_cluster/_mgmtdb0.log" for further details.
9. Finally run mgmtca and verify mgmtdb resoruce is available
run mgmtca

ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
10. As mentioned in upgarde to 12.2 the SCAN name changes if GNS is used. This cluster uses GNS. Check if the SCAN name has been reverted back to 12.1 format.
srvctl config scan
SCAN name: prod-cluster-scan.rac.domain.net, Network: 1
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
SCAN 0 IPv4 VIP: -/scan1-vip/192.168.0.92
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 1 IPv4 VIP: -/scan2-vip/192.168.0.89
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: -/scan3-vip/192.168.0.90
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
This concludes the downgrade to 12.1.0.2 from 12.2.0.1

Related posts
Downgrade Grid Infrastructure from 12.2.0.1 to 11.2.0.4
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7

Useful metalink notes
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]

Sunday, October 15, 2017

Upgrading RAC from 12.1.0.2 to 12.2.0.1 - Database (non-CDB and CDB)

After upgrading the GI from 12.1.0.2 to 12.2.0.1 the next phase is to upgrade the database software and the actual databases. This cluster has two standard edition 2 RAC databases. One is a non-CDB while other is a CDB with just one PDB. The post shows upgrade of them of both. First is the database software upgrade. This is done as an out of place upgrade. Before upgrade commence run cluvfy with database pre-upgrade option.
[oracle@rhel12c1 ~]$ /opt/app/12.2.0/grid/bin/cluvfy stage  -pre dbinst -upgrade -src_dbhome /opt/app/oracle/product/12.1.0/dbhome_2 -dest_dbhome /opt/app/oracle/product/12.2.0/dbhome_1 -dest_version 12.2.0.1.0
12.2 has introduced a new OS group for RAC management. Add the group to oracle user, keeping the user equivalence across all nodes.
# groupadd racdba
# usermod -g oinstall -G dba,oper,asmdba,asmoper,asmadmin,backupdba,dgdba,kmdba,racdba oracle
# id oracle
uid=500(oracle) gid=502(oinstall) groups=502(oinstall),501(dba),503(oper),504(asmoper),505(asmdba),506(asmadmin),507(backupdba),508(dgdba),509(kmdba),510(racdba)
Run 12.2 installer and select software only install option.

Upgrading non-CDB

As mentioned earlier this cluster has two databases, non-CDB and CDB. This section shows the upgrading of the non-CDB. Before the upgrade make sure the sga_target is at least 2068MB. This requirement is not flagged in orachk or any other pre-req checks. If sga_target is lower than this, the upgrade would fail.
The component version and status before the upgrade is given below
COMP_ID    COMP_NAME                           VERSION    STATUS
---------- ----------------------------------- ---------- --------
CATALOG    Oracle Database Catalog Views       12.1.0.2.0 VALID
CATPROC    Oracle Database Packages and Types  12.1.0.2.0 VALID
RAC        Oracle Real Application Clusters    12.1.0.2.0 VALID
XDB        Oracle XML Database                 12.1.0.2.0 VALID
OWM        Oracle Workspace Manager            12.1.0.2.0 VALID
CONTEXT    Oracle Text                         12.1.0.2.0 VALID



Run the preupgrade.jar, the new pre-upgrade check tool available with 12.2 for pre-upgrade checks.
[oracle@rhel12c1 ~]$ $ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STD12C1
     Container Name:  STD12C1
       Container ID:  0
            Version:  12.1.0.2.0
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  SE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             550 MB  ENABLED       667 MB  None
     SYSTEM                             700 MB  ENABLED       806 MB  None
     TEMP                                21 MB  ENABLED       150 MB  None
     UNDOTBS1                           230 MB  ENABLED       400 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.

   + (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

     The database contains 179 objects in the recycle bin.

     The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
   + Run 12.1.0.2.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     8 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + Review and remove any unnecessary EVENTS.

     The database contains events.

     There are events set that should be removed before upgrade, unless your
     application vendors and/or Oracle Support state differently.  Changes
     will need to be made in the spfile.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 12.1.0.2
     Oracle Database SQL Tuning Guide.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 18 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.


Preupgrade generated files:
    /opt/app/oracle/cfgtoollogs/std12c1/preupgrade/preupgrade_fixups.sql
    /opt/app/oracle/cfgtoollogs/std12c1/preupgrade/postupgrade_fixups.sql
Run the pre-upgrade fixup
SQL> @/opt/app/oracle/cfgtoollogs/std12c1/preupgrade/preupgrade_fixups.sql

Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 12.2.0.1.0 Build: 1
Generated on:            2017-09-19 16:31:13

For Source Database:     STD12C1
Source Database Version: 12.1.0.2.0
For Upgrade to Version:  12.2.0.1.0

                          Fixup
Check Name                Status  Further DBA Action
----------                ------  ------------------
purge_recyclebin          Passed  None
invalid_objects_exist     Failed  Manual fixup recommended.
underscore_events         Failed  Manual fixup recommended.
dictionary_stats          Passed  None

PL/SQL procedure successfully completed.
Once the pre-ugprade checks are fixed run dbua from the 12.2 Oracle home. Select the database for upgrade, in this case the non-CDB is selected.
Timezone is upgraded at the same time as database.
Upgrade summary
Upgrade progress
Upgrade completion summary
Run the postupgrade fixup script
SQL> @/opt/app/oracle/cfgtoollogs/std12c1/preupgrade/postupgrade_fixups.sql

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 12.2.0.1.0 Build: 1
Generated on:            2017-09-19 16:35:42

For Source Database:     STD12C1
Source Database Version: 12.1.0.2.0
For Upgrade to Version:  12.2.0.1.0

                          Fixup
Check Name                Status  Further DBA Action
----------                ------  ------------------
old_time_zones_exist      Passed  None
post_dictionary           Passed  None
After the upgrade the DB components version and status
COMP_ID    COMP_NAME                           VERSION    STATUS
---------- ----------------------------------- ---------- --------
CATALOG    Oracle Database Catalog Views       12.2.0.1.0 VALID
CATPROC    Oracle Database Packages and Types  12.2.0.1.0 VALID
RAC        Oracle Real Application Clusters    12.2.0.1.0 VALID
XDB        Oracle XML Database                 12.2.0.1.0 VALID
OWM        Oracle Workspace Manager            12.2.0.1.0 VALID
CONTEXT    Oracle Text                         12.2.0.1.0 VALID
The database registry history added an entry for the upgrade
ACTION_TIME                    ACTION          NAMESPACE  VERSION            ID COMMENTS
------------------------------ --------------- ---------- ---------- ---------- ----------------------------------------
                               BOOTSTRAP       DATAPATCH  12.1.0.2              RDBMS_12.1.0.2.0DBPSU_LINUX.X64_161210
                               BOOTSTRAP       DATAPATCH  12.2.0.1              RDBMS_12.2.0.1.0_LINUX.X64_170125
19-SEP-17 05.18.47.947470 PM   VIEW INVALIDATE                          8289601 view invalidation
19-SEP-17 05.20.49.419414 PM   UPGRADE         SERVER     12.2.0.1.0            Upgraded from 12.1.0.2.0
Verify the timezone upgrade
SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_26.dat              26          0
This SE2 DB had statspack installed. Upgrade the statspack to 12.2
SQL>@?/rdbms/admin/spup12102.sql
SQL>@?/rdbms/admin/spup12200.sql
The remote_listener parameter had the new scan name added to it automatically. It also included the old scan name as well.
remote_listener  string      prod-cluster-scan.rac.domain.net:1521, prod-cluster-scan.prod-cluster.rac.domain.net:1521
Finally upgrade the database compatibility parameter
SQL> alter system set compatible='12.2.0.1.0' scope=spfile sid='*';
Upgrading CDB

The component version and status on the CDB before the upgrade is as below
    CON_ID COMP_ID    COMP_NAME                           VERSION         STATUS
---------- ---------- ----------------------------------- --------------- ----------
         1 APEX       Oracle Application Express          4.2.5.00.08     VALID
         1 CATALOG    Oracle Database Catalog Views       12.1.0.2.0      VALID
         1 CATJAVA    Oracle Database Java Packages       12.1.0.2.0      VALID
         1 CATPROC    Oracle Database Packages and Types  12.1.0.2.0      VALID
         1 CONTEXT    Oracle Text                         12.1.0.2.0      VALID
         1 JAVAVM     JServer JAVA Virtual Machine        12.1.0.2.0      VALID
         1 ORDIM      Oracle Multimedia                   12.1.0.2.0      VALID
         1 OWM        Oracle Workspace Manager            12.1.0.2.0      VALID
         1 RAC        Oracle Real Application Clusters    12.1.0.2.0      VALID
         1 XDB        Oracle XML Database                 12.1.0.2.0      VALID
         1 XML        Oracle XDK                          12.1.0.2.0      VALID

         3 APEX       Oracle Application Express          4.2.5.00.08     VALID
         3 CATALOG    Oracle Database Catalog Views       12.1.0.2.0      VALID
         3 CATJAVA    Oracle Database Java Packages       12.1.0.2.0      VALID
         3 CATPROC    Oracle Database Packages and Types  12.1.0.2.0      VALID
         3 CONTEXT    Oracle Text                         12.1.0.2.0      VALID
         3 JAVAVM     JServer JAVA Virtual Machine        12.1.0.2.0      VALID
         3 ORDIM      Oracle Multimedia                   12.1.0.2.0      VALID
         3 OWM        Oracle Workspace Manager            12.1.0.2.0      VALID
         3 RAC        Oracle Real Application Clusters    12.1.0.2.0      VALID
         3 XDB        Oracle XML Database                 12.1.0.2.0      VALID
         3 XML        Oracle XDK                          12.1.0.2.0      VALID
Similar to CDB run the preupgrade.jar from console. The output will have a section for each container (root, pdb see and all the pdbs opened).
[oracle@rhel12c1 ~]$ $ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STDCDB
     Container Name:  CDB$ROOT
       Container ID:  1
            Version:  12.1.0.2.0
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  SE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  JServer JAVA Virtual Machine           [to be upgraded]  VALID
  Oracle XDK for Java                    [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID
  Oracle Java Packages                   [to be upgraded]  VALID
  Oracle Multimedia                      [to be upgraded]  VALID
  Oracle Application Express             [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups_CDB_ROOT.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             850 MB  ENABLED      1690 MB  None
     SYSTEM                             780 MB  ENABLED      1274 MB  None
     TEMP                                72 MB  ENABLED       150 MB  None
     UNDOTBS2                           370 MB  ENABLED       400 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.


   + (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

     The database contains 62 objects in the recycle bin.

     The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
   + Run 12.1.0.2.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     7 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 12.1.0.2
     Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
   + Consider upgrading APEX manually, before the database upgrade.

     The database contains APEX version 4.2.5.00.08 and will need to be
     upgraded to at least version 5.0.4.00.12.

     To reduce database upgrade time, you can upgrade APEX manually before
     the database upgrade.  Refer to My Oracle Support Note 1088970.1 for
     information on APEX installation upgrades.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups_CDB_ROOT.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 18 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.

Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STDCDB
     Container Name:  PDB$SEED
       Container ID:  2
            Version:  12.1.0.2.0
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  TRUE
            Edition:  SE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  JServer JAVA Virtual Machine           [to be upgraded]  VALID
  Oracle XDK for Java                    [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID
  Oracle Java Packages                   [to be upgraded]  VALID
  Oracle Multimedia                      [to be upgraded]  VALID
  Oracle Application Express             [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups_PDB_SEED.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             495 MB  ENABLED      1384 MB  None
     SYSTEM                             270 MB  ENABLED       759 MB  None
     TEMP                                62 MB  ENABLED       150 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.

  RECOMMENDED ACTIONS
  ===================
   + Run 12.1.0.2.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     11 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 12.1.0.2
     Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
   + Consider upgrading APEX manually, before the database upgrade.

     The database contains APEX version 4.2.5.00.08 and will need to be
     upgraded to at least version 5.0.4.00.12.

     To reduce database upgrade time, you can upgrade APEX manually before
     the database upgrade.  Refer to My Oracle Support Note 1088970.1 for
     information on APEX installation upgrades.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups_PDB_SEED.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 18 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.


Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STDCDB
     Container Name:  STDPDB
       Container ID:  3
            Version:  12.1.0.2.0
         Compatible:  12.1.0.2.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  18
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  SE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  JServer JAVA Virtual Machine           [to be upgraded]  VALID
  Oracle XDK for Java                    [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID
  Oracle Java Packages                   [to be upgraded]  VALID
  Oracle Multimedia                      [to be upgraded]  VALID
  Oracle Application Express             [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups_STDPDB.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             525 MB  ENABLED      1411 MB  None
     SYSTEM                             290 MB  ENABLED       785 MB  None
     TEMP                                62 MB  ENABLED       150 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.

   + (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

     The database contains 16 objects in the recycle bin.

     The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
   + Run 12.1.0.2.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     85 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 12.1.0.2
     Oracle Database SQL Tuning Guide.

  INFORMATION ONLY
  ================
   + Consider upgrading APEX manually, before the database upgrade.

     The database contains APEX version 4.2.5.00.08 and will need to be
     upgraded to at least version 5.0.4.00.12.

     To reduce database upgrade time, you can upgrade APEX manually before
     the database upgrade.  Refer to My Oracle Support Note 1088970.1 for
     information on APEX installation upgrades.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups_STDPDB.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 18 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.


Preupgrade generated files:
    /opt/app/oracle/cfgtoollogs/stdcdb/preupgrade/preupgrade_fixups.sql
    /opt/app/oracle/cfgtoollogs/stdcdb/preupgrade/postupgrade_fixups.sql
As seen from output above each section has it's own pre and post upgrade script. However the final pre and post upgrade scripts (preupgrade_fixups.sql and postupgrade_fixups.sql) mentioned have the same content as those individual container specific scripts. Moreover they have container specif sections that will only run on the specified container. These *upgrade_fixups scripts could be run in all the containers using catcon.pl as below.
$ cd /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/admin/
$  $ORACLE_HOME/perl/bin/perl catcon.pl -n 1 -e -b preupgrade_fixups /opt/app/oracle/cfgtoollogs/stdcdb/preupgrade/preupgrade_fixups.sql
catcon: ALL catcon-related output will be written to preupgrade_fixups_catcon_30484.lst
catcon: See preupgrade_fixups*.log files for output generated by scripts
catcon: See preupgrade_fixups_*.lst files for spool files, if any
catcon.pl: completed successfully
Once the preupgrade scripts have completed, run dbua to begin CDB upgrade.
Select the PDBs upgraded at the same time as CDB. As this is a SE2 only single PDB is allwoed and that single PDB is upgraded along with the CDB.
The other steps are similar to that of non-CDB. Below is the pre-upgrade summary
Upgrade progress
Post upgrade summary
Run the post upgrade script on all containers using catcon.pl
 $ORACLE_HOME/perl/bin/perl catcon.pl -n 1 -e -b postupgrade_fixups /opt/app/oracle/cfgtoollogs/stdcdb/preupgrade/postupgrade_fixups.sql
catcon: ALL catcon-related output will be written to [/opt/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/postupgrade_fixups_catcon_11556.lst]
catcon: See [/opt/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/postupgrade_fixups*.log] files for output generated by scripts
catcon: See [/opt/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/postupgrade_fixups*.lst] files for spool files, if any

catcon.pl: completed successfully
Component status and version after the upgrade
    CON_ID COMP_ID         COMP_NAME                           STATUS   VERSION
---------- --------------- ----------------------------------- -------- ---------------
         1 CATALOG         Oracle Database Catalog Views       VALID    12.2.0.1.0
         1 CATPROC         Oracle Database Packages and Types  VALID    12.2.0.1.0
         1 JAVAVM          JServer JAVA Virtual Machine        VALID    12.2.0.1.0
         1 XML             Oracle XDK                          VALID    12.2.0.1.0
         1 CATJAVA         Oracle Database Java Packages       VALID    12.2.0.1.0
         1 RAC             Oracle Real Application Clusters    VALID    12.2.0.1.0
         1 XDB             Oracle XML Database                 VALID    12.2.0.1.0
         1 OWM             Oracle Workspace Manager            VALID    12.2.0.1.0
         1 CONTEXT         Oracle Text                         VALID    12.2.0.1.0
         1 ORDIM           Oracle Multimedia                   VALID    12.2.0.1.0
         1 APEX            Oracle Application Express          VALID    5.0.4.00.12

         3 CATALOG         Oracle Database Catalog Views       VALID    12.2.0.1.0
         3 CATPROC         Oracle Database Packages and Types  VALID    12.2.0.1.0
         3 JAVAVM          JServer JAVA Virtual Machine        VALID    12.2.0.1.0
         3 XML             Oracle XDK                          VALID    12.2.0.1.0
         3 CATJAVA         Oracle Database Java Packages       VALID    12.2.0.1.0
         3 RAC             Oracle Real Application Clusters    VALID    12.2.0.1.0
         3 XDB             Oracle XML Database                 VALID    12.2.0.1.0
         3 OWM             Oracle Workspace Manager            VALID    12.2.0.1.0
         3 CONTEXT         Oracle Text                         VALID    12.2.0.1.0
         3 ORDIM           Oracle Multimedia                   VALID    12.2.0.1.0
         3 APEX            Oracle Application Express          VALID    5.0.4.00.12
Database registry history for each container
    CON_ID ACTION          NAMESPACE  VERSION         COMMENTS
---------- --------------- ---------- --------------- ----------------------------------------
         1 BOOTSTRAP       DATAPATCH  12.1.0.2        RDBMS_12.1.0.2.0DBPSU_LINUX.X64_161210
         1 BOOTSTRAP       DATAPATCH  12.2.0.1        RDBMS_12.2.0.1.0_LINUX.X64_170125
         1 VIEW INVALIDATE                            view invalidation
         1 UPGRADE         SERVER     12.2.0.1.0      Upgraded from 12.1.0.2.0

         3 BOOTSTRAP       DATAPATCH  12.1.0.2        RDBMS_12.1.0.2.0DBPSU_LINUX.X64_161210
         3 BOOTSTRAP       DATAPATCH  12.2.0.1        RDBMS_12.2.0.1.0_LINUX.X64_170125
         3 VIEW INVALIDATE                            view invalidation
         3 UPGRADE         SERVER     12.2.0.1.0      Upgraded from 12.1.0.2.0
Check the timezone on root and PDB
SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_26.dat              26          0

SQL> alter session set container=stdpdb;
SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_26.dat              26          0
If statspack is configured on PDB then upgrade it 12.2. Change the line
connect perfstat/&&perfstat_password
to
connect perfstat/&&perfstat_password@&&tns_alias
in both spup12102.sql and spup12200.sql. Then create a script with below content (replace password and tns alias values as needed) and run with catcon.pl. (Refer 2020285.1 for more)
cat statup.sql
define perfstat_password='perfstat'
define tns_alias='stdpdb'
@?/rdbms/admin/spup12102.sql;
@?/rdbms/admin/spup12200.sql;

$ORACLE_HOME/perl/bin/perl catcon.pl -n 1 -e -b statup -c stdpdb statup.sql
Same as CDB the remote_listener parameter had the new scan name added to it automatically. It also had the old scan name as well.
remote_listener  string      prod-cluster-scan.rac.domain.net:1521, prod-cluster-scan.prod-cluster.rac.domain.net:1521
Finally upgrade the compatible parameter on the CDB.
SQL> alter system set compatible='12.2.0.1.0' scope=spfile sid='*';
That concludes the upgrading of the databases.

Now that all the databases in the cluster are upgraded, increase the ASM related compatible parameters to 12.2
alter diskgroup fra SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup DATA  SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup fra set attribute 'compatible.rdbms'='12.2.0.1.0';
alter diskgroup data set attribute 'compatible.rdbms'='12.2.0.1.0';
alter diskgroup clusfs  SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup clusfs set attribute 'compatible.rdbms'='12.2.0.1.0';
alter diskgroup GIMR  SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup GIMR set attribute 'compatible.rdbms'='12.2.0.1.0';

GROUP_NUMBER NAME                      VALUE
------------ ------------------------- ---------------
           1 compatible.asm            12.2.0.1.0
           1 compatible.rdbms          12.2.0.1.0
           2 compatible.asm            12.2.0.1.0
           2 compatible.rdbms          12.2.0.1.0
           3 compatible.asm            12.2.0.1.0
           3 compatible.rdbms          12.2.0.1.0
           4 compatible.asm            12.2.0.1.0
           4 compatible.rdbms          12.2.0.1.0
As mentioned in previous post, upgrade adds new oracle user accounts with default passwords. Change the default passwords.

Finally run
orachk -u -o post
to check post upgrade status.

Related Posts
Upgrading RAC from 11.2.0.4 to 12.2.0.1 - Database
Upgrading 11.2.0.3 (11gR2) Database to 12.1.0.1 (12c) Using DBUA
Upgrading from 11.2.0.3 to 12.1.0.1 RAC
Upgrade Oracle Database 12c1 from 12.1.0.1 to 12.1.0.2
Upgrading 12c CDB and PDB from 12.1.0.1 to 12.1.0.2
Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Database
Upgrading RAC from 12.1.0.1 to 12.1.0.2 - Database
Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Database