Friday, May 3, 2019

Upgrading Oracle Restart from 18c (18.6) to 19c (19.3)

This post lists the steps for upgrading an Oracle restart environment (Single instance non-CDB on ASM) from 18.6 to 19.3. The 18c setup is on RHEL 7. The OS and kernel versions are as follows
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)

uname -r
3.10.0-957.el7.x86_64
The current versions of the GI are
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [18.0.0.0.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [18.0.0.0.0]
The Oracle restart environment used is not a role separated setup, both Oracle and GI homes are installed using oracle user.
Download and run orachk -u -o pre (1268927.2) on the 18c installation to identify any required patches before the upgrade. With the above 18.6 RU applied no other patches were required be applied prior to the upgrade.
19c grid software instillation is also similar to 18c and is based on a image file. Before unzipping the GI image file create the 19c GI home directory and then unzip the GI image file into it.
mkdir -p /opt/app/oracle/product/19.x.0/grid
unzip LINUX.X64_193000_grid_home.zip -d /opt/app/oracle/product/19.x.0/grid
Use cluvfy with hacfg to verify pre-reqs. The SWAP size failure shown is ignorable on a test system.
cd /opt/app/oracle/product/19.x.0/grid/

./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: ip-172-31-2-77:/usr,ip-172-31-2-77:/var,ip-172-31-2-77:/etc,ip-172-31-2-77:/sbin,ip-172-31-2-77:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying Swap Size ...FAILED
ip-172-31-2-77: PRVF-7573 : Sufficient swap size is not available on node
                "ip-172-31-2-77" [Required = 16GB (1.6777216E7KB) ; Found = 0.0
                bytes]
ASM will be upgraded as part of the upgrade process.
Therefore stop the database before starting the GI upgrade.
srvctl stop database -d gold
Start the GI upgrade by running the gridSetup.sh from the grid home.
cd /opt/app/oracle/product/19.x.0/grid
./gridSetup.sh
Select upgrade GI option.
Location of the GI home cannot be changed.
Upgrade summary page.
Run root upgrade script when prompted.
Following shows the output from running the rootugprade script.
# /opt/app/oracle/product/19.x.0/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/roothas_2019-05-02_11-06-31AM.log
2019/05/02 11:06:34 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2019/05/02 11:06:37 CLSRSC-363: User ignored prerequisites during installation
2019/05/02 11:06:37 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2019/05/02 11:06:41 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2019/05/02 11:06:41 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2019/05/02 11:06:41 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.

ASM has been upgraded and started successfully.

2019/05/02 11:07:21 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2019/05/02 11:07:22 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
2019/05/02 11:07:26 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-2-77 successfully pinned.
2019/05/02 11:07:29 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2019/05/02 11:07:30 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2019/05/02 11:07:30 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2019/05/02 11:07:50 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.

ip-172-31-2-77     2019/05/02 07:08:28     /opt/app/oracle/crsdata/ip-172-31-2-77/olr/backup_20190502_070828.olr     724960844

ip-172-31-2-77     2019/05/02 05:23:49     /opt/app/oracle/product/18.x.0/grid/cdata/ip-172-31-2-77/backup_20190502_052349.olr     2532936542
2019/05/02 11:08:28 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
2019/05/02 11:10:08 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
Click OK on the execution configuration to proceed with the rest of the upgrade steps. Following shows the end of the upgrade page.
The HAS software is now upgraded to 19c.
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [19.0.0.0.0]
If the ASM spfile was moved to $GI_HOME/dbs (described in step 12 in an earlier post) then this need to be moved to 19c GI_HOME.
ASMCMD> spget
/opt/app/oracle/product/18.x.0/grid/dbs/spfile+ASM.ora
ASMCMD> spcopy /opt/app/oracle/product/18.x.0/grid/dbs/spfile+ASM.ora  /opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
ASMCMD> spset /opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
ASMCMD> spget
/opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
Use cluvfy post check validate as follows
cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.

CVU operation performed:      stage -post hacfg
Date:                         May 2, 2019 11:43:30 AM
CVU home:                     /opt/app/oracle/product/19.x.0/grid/
User:                         oracle


Next step is to upgrade the database software. Similar to GI home instillation, this too is based on image. Create the 19c oracle home location and unzip the DB software image file into it.
mkdir -p /opt/app/oracle/product/19.x.0/dbhome_1
unzip LINUX.X64_193000_db_home.zip -d /opt/app/oracle/product/19.x.0/dbhome_1
To begin the installation execute runInstaller from Oracle home location.
cd /opt/app/oracle/product/19.x.0/dbhome_1
./runInstaller
Select software only setup for the instillation option.
Select single database installation as this is an Oracle restart setup.
Select appropriate edition based on the licensing.
What is interesting in the above is that SE2 no longer mentions RAC usages where as in 18c it did. With 19c, RAC is not an option available under SE2.
Oracle home location is fixed and the location is where the DB software was unzipped.
There are no new OS groups introduced in 19c. So all the OS groups used 18c are used here as well.
DB software installation summary.
When prompted run the root script. Unlike 18c there's no prompt for installing TFA. A standalone TFA is setup as part of 19c.
# /opt/app/oracle/product/19.x.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.x.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Oracle Trace File Analyzer (TFA - Standalone Mode) is available at :
    /opt/app/oracle/product/19.x.0/dbhome_1/bin/tfactl

Note :
1. tfactl will use TFA Service if that service is running and user has been granted access
2. tfactl will configure TFA Standalone Mode only if user has no access to TFA Service or TFA is not installed


After the database software is installed next step is upgrade of the database. 19c has the option of autoupgrade (which is also backported to 12.2 and 18c) which automates the upgrade process. However, in this post the DB will be upgraded using the DBUA. Run preupgrade.jar available in the 19c home (19c_home/rdbms/admin) to check the upgrade readiness of the database (for more refer 2421552.1)
 $ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 1 on 2019-05-02T12:16:42

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  GOLD
     Container Name:  gold
       Container ID:  0
            Version:  18.0.0.0.0
     DB Patch Level:  Database Release Update : 18.6.0.0.190416 (29301631)
         Compatible:  18.0.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  31
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  1.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 18.0.0.0
      Oracle Database Upgrade Guide.

  2.  (AUTOFIXUP) Gather statistics on fixed objects prior the upgrade.

      None of the fixed object tables have had stats collected.

      Gathering statistics on fixed objects, if none have been gathered yet, is
      recommended prior to upgrading.

      For information on managing optimizer statistics, refer to the 18.0.0.0
      Oracle Database Upgrade Guide.

  INFORMATION ONLY
  ================
  3.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      TEMP                                20 MB       150 MB
      UNDOTBS1                           365 MB       412 MB

      Minimum tablespace sizes for upgrade are estimates.

  4.  Check the Oracle Backup and Recovery User's Guide for information on how
      to manage an RMAN recovery catalog schema.

      If you are using a version of the recovery catalog schema that is older
      than that required by the RMAN client version, then you must upgrade the
      catalog schema.

      It is good practice to have the catalog schema the same or higher version
      than the RMAN client version you are using.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database GOLD
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  5.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 31 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  6.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  7.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 18.0.0.0
      Oracle Database Upgrade Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database GOLD
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/gold/preupgrade/postupgrade_fixups.sql


==================
PREUPGRADE SUMMARY
==================
  /opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade.log
  /opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade_fixups.sql
  /opt/app/oracle/cfgtoollogs/gold/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/opt/app/oracle/cfgtoollogs/gold/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2019-05-02T12:16:42
Run the pre-upgrade fixup script and then DBUA from the 19c home. Select the database to upgrade.
Check and resolve any validation concerns.
Parallel upgrade will speed up the upgrade process. This step also allows to specify whether timezone is also upgraded at the same time as the database.
The upgrade summary page.
Upgrade progress.
Upgrade result page.
Once the upgrade has finished execute the post-upgrade fixup script (mentioned in the output when preupgrade.jar was run). The DB components versions and status after the upgrade is as follows.
SQL> select comp_name,status,version,version_full from cdb_registry order by 1,2;

COMP_NAME                                STATUS          VERSION         VERSION_FULL
---------------------------------------- --------------- --------------- ---------------
Oracle Database Catalog Views            VALID           19.0.0.0.0      19.3.0.0.0
Oracle Database Packages and Types       VALID           19.0.0.0.0      19.3.0.0.0
Oracle Real Application Clusters         OPTION OFF      19.0.0.0.0      19.3.0.0.0
Oracle Text                              VALID           19.0.0.0.0      19.3.0.0.0
Oracle Workspace Manager                 VALID           19.0.0.0.0      19.3.0.0.0
Oracle XML Database                      VALID           19.0.0.0.0      19.3.0.0.0

6 rows selected.
The timezone file has been upgraded to 32.
SQL> select * from v$timezone_file;

FILENAME                     VERSION     CON_ID
-------------------- --------------- ----------
timezlrg_32.dat                   32          0
If satisfied with the upgrade and application testing then change the compatibility parameter on the DB and ASM disk groups.
SQL> alter system set compatible='19.0.0' scope=spfile;
shutdown immediate;
Login as grid user and then login to ASM instance as sysasm to change asm compatibility parameters.
SQL> alter diskgroup FRA SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup DATA  SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup fra set attribute 'compatible.rdbms'='19.0.0.0.0';
SQL> alter diskgroup data set attribute 'compatible.rdbms'='19.0.0.0.0';

SQL> select g.name,a.name,a.value from v$asm_diskgroup g, v$asm_attribute a where g.group_number=a.group_number and a.name like '%compat%';

NAME                 NAME                 VALUE
-------------------- -------------------- --------------------
DATA                 compatible.asm       19.0.0.0.0
DATA                 compatible.rdbms     19.0.0.0.0
FRA                  compatible.asm       19.0.0.0.0
FRA                  compatible.rdbms     19.0.0.0.0
After ASM compatibility parameters are updated start the database.
Finally run orachk -u -o post check the post upgrade state of the oracle restart setup.

Related Posts
Upgrading Oracle Restart from 12.2.0.1 to 18c
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2