Upgrade from 19c to 21c Using Autoupgrade

This post shows the steps for upgrading a 19c database (CDB + PDB) with autoupgrade tool. There's an earlier post which shows upgrading PDB form 19c to 21 using unplug-plug-upgrade method.
If any privileges were reovked from public due to security reason (CIS standard recommendations) then grant those privileges back before starting the ugprade process. Once the upgrade is completed then those privilges could be revoked again.
Generate a sample upgrade config file with

java -jar autoupgrade.jar -create_sample_file config normalupgrade.cfg

and customize the "Full DB/CDB upgrade" section.

global.autoupg_log_dir=/home/oracle/autoupgrade

#
# Database number 1 - Full DB/CDB upgrade
#
upg1.log_dir=/home/oracle/autoupgrade/testcdb             # Path of the log directory for the upgrade job
upg1.sid=testcdb                                              # ORACLE_SID of the source DB/CDB
upg1.source_home=/opt/app/oracle/product/19.x.0/dbhome_2  # Path of the source ORACLE_HOME
upg1.target_home=/opt/app/oracle/product/21.x.0/dbhome_1  # Path of the target ORACLE_HOME
upg1.start_time=NOW                                       # Optional. [NOW | +XhYm (X hours, Y minutes after launch) | dd/mm/yyyy hh:mm:ss]
upg1.upgrade_node=ip-172-31-10-91.eu-west-1.compute.internal                                # Optional. To find out the name of your node, run the hostname utility. Default is 'localhost'
upg1.run_utlrp=yes                                  # Optional. Whether or not to run utlrp after upgrade
upg1.timezone_upg=yes                               # Optional. Whether or not to run the timezone upgrade
upg1.target_version=21                      # Oracle version of the target ORACLE_HOME.  Only required when the target Oracle database version is 12.2

The databae being upgraded is called testcdb. The source home is the 19c home and the target home is the 21c home. DB timezone is also upgraded the same time.
Run the autoupgrade mode in analyze mode to check the ugprade readiness of the database.

java -jar autoupgrade.jar -config normalupgrade.cfg -mode analyze

This will generate a status report (status.html) at the end. Cheeck for any pre-req work that need to be carried out manually before the upgrade.

Run autougprade in fixup mode to execute preupgrade fixes.

java -jar autoupgrade.jar -config normalupgrade.cfg -mode fixups

Finally run the autoupgrade in deploy mode to commence the upgrade.

java -jar autoupgrade.jar -config normalupgrade.cfg -mode deploy

If upgrade is successful then drop the GRP associated with it. It is important that GRP is dropped before increasing the compatibility.
To change the compatibility update the compatibility parameter and restart the database.

SQL> alter system set compatible='21.0.0' scope=spfile;

Following will be shown in the alert log.

ALERT: Compatibility of the database is changed from 19.0.0.0.0 to 21.0.0.0.0.
Increased the record size of controlfile section 15 to 104 bytes
Control file expanded from 1156 blocks to 1158 blocks

One of the postupgrade tasks is to run the $ORACLE_HOME/rdbms/admin/auditpostupgrade.sql script (mentioned in 2659172.1 as well). However, this script is not available in 21c home (checked on 21.7 and 21.8 the last RU at the time of this post). After raising an SR oracle support confirmed that script is there on 23c and at the moment not avialable on 21c. So for the time being to ignore the execution of auditpostupgrade.sql script.

Related Posts
Upgrading Oracle Restart from 19c to 21c on RHEL 7
Unplug-Plug-Upgrade from 19c to 21c Using Autoupgrade

Out of Place (OOP) Patching of Oracle Restart

Oracle grid infrastructure deployed in a RAC configuration has the option switchGridHome for out of place patching. But this option doesn't work with Oracle restart.

./gridSetup.sh -silent -switchGridHome -applyRU /opt/app/oracle/installs/19.19/35037840
Preparing the home to patch...
Preparing the home to apply the patch failed. For details look at the logs from /opt/app/oraInventory/logs.
The log can be found at: /opt/app/oraInventory/logs/GridSetupActions2023-05-11_10-17-15AM/installerPatchActions_2023-05-11_10-17-15AM.log
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-45101] Clusterware is not running on the local node.
   ACTION: Ensure that the Clusterware is configured and is running on local node before proceeding.

MOS Doc 2764906.1 states that switchGridHome option is not supported for Oracle Restart.
However, there is a way to do OOP on Oracle Restart explained here.
This post is based on OOP of a Oracle restart using the steps mentioned in the link above. The current configuration consists of following resources

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-2-77
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-2-77
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-2-77
ora.asm                   ora.asm.type              ONLINE             ONLINE             ip-172-31-2-77
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-2-77
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-2-77
ora.ons                   ora.ons.type              ONLINE             ONLINE             ip-172-31-2-77
ora.testcdb.db            ora.database.type         ONLINE             ONLINE             ip-172-31-2-77
ora.testcdb.dbxrw.svc     ora.service.type          ONLINE             ONLINE             ip-172-31-2-77
ora.testcdb.testsrv.svc   ora.service.type          ONLINE             ONLINE             ip-172-31-2-77

The current GI Home is /opt/app/oracle/product/19.x.0/grid
The new GI home will be /opt/app/oracle/product/19.19.0/grid
1. First step is to install new GI home with the required patches using the software only option. How to do a software only Oracle restart installation was shown in a previous post. In this instance a response file is used and 19.19 RU is applied at install time.

$ ./gridSetup.sh -silent -responseFile /opt/app/oracle/installs/19.19/grid_sw_only.rsp -applyRU /opt/app/oracle/installs/19.19/35037840
Preparing the home to patch...
Applying the patch /opt/app/oracle/installs/19.19/35037840...
Successfully applied the patch.
The log can be found at: /opt/app/oraInventory/logs/GridSetupActions2023-05-11_11-37-14AM/installerPatchActions_2023-05-11_11-37-14AM.log
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-32022] Grid infrastructure software for a cluster installation must not be under an Oracle base directory.
   CAUSE: Grid infrastructure for a cluster installation assigns root ownership to all parent directories of the Grid home location. As a result, ownership of all named directories in the software location path is changed to root, creating permissions errors for all subsequent installations into the same Oracle base.
   ACTION: Specify software location outside of an Oracle base directory for grid infrastructure for a cluster installation.
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /opt/app/oraInventory/logs/GridSetupActions2023-05-11_11-37-14AM/gridSetupActions2023-05-11_11-37-14AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /opt/app/oraInventory/logs/GridSetupActions2023-05-11_11-37-14AM/gridSetupActions2023-05-11_11-37-14AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /opt/app/oracle/product/19.19.0/grid/install/response/grid_2023-05-11_11-37-14AM.rsp

You can find the log of this install session at:
 /opt/app/oraInventory/logs/GridSetupActions2023-05-11_11-37-14AM/gridSetupActions2023-05-11_11-37-14AM.log

As a root user, execute the following script(s):
        1. /opt/app/oracle/product/19.19.0/grid/root.sh

Execute /opt/app/oracle/product/19.19.0/grid/root.sh on the following nodes:
[ip-172-31-2-77]

Successfully Setup Software with warning(s).

Run the root.sh

/opt/app/oracle/product/19.19.0/grid/root.sh
Check /opt/app/oracle/product/19.19.0/grid/install/root_ip-172-31-2-77.eu-west-1.compute.internal_2023-05-11_11-52-11-416203678.log for the output of root script

# more /opt/app/oracle/product/19.19.0/grid/install/root_ip-172-31-12-6.eu-west-1.compute.internal_2023-05-11_11-52-11-416203678.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.19.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Cluster or Grid Infrastructure for a Stand-Alone Server execute the following command as oracle user:
/opt/app/oracle/product/19.19.0/grid/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

2. Verify the new GI home has the patches applied.

export ORACLE_HOME=/opt/app/oracle/product/19.19.0/grid
$ORACLE_HOME/OPatch/opatch lspatches
35107512;TOMCAT RELEASE UPDATE 19.0.0.0.0 (35107512)
35050331;OCW RELEASE UPDATE 19.19.0.0.0 (35050331)
35050325;ACFS RELEASE UPDATE 19.19.0.0.0 (35050325)
35042068;Database Release Update : 19.19.0.0.230418 (35042068)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

OPatch succeeded.

3. As root run the prepatch steps on the new GI home. This does not bring any of the running services down.

# /opt/app/oracle/product/19.19.0/grid/crs/install/roothas.sh -prepatch -dstcrshome /opt/app/oracle/product/19.19.0/grid
Using configuration parameter file: /opt/app/oracle/product/19.19.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/hapatch_2023-05-11_11-48-38AM.log
2023/05/11 11:48:59 CLSRSC-347: Successfully unlock /opt/app/oracle/product/19.19.0/grid
2023/05/11 11:48:59 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.

4. As root run the postpatch step on the new GI home. This step results in currently running services being brought down and started using the new GI home.

# /opt/app/oracle/product/19.19.0/grid/crs/install/roothas.sh -postpatch -dstcrshome /opt/app/oracle/product/19.19.0/grid
Using configuration parameter file: /opt/app/oracle/product/19.19.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/hapatch_2023-05-11_11-49-46AM.log
Redirecting to /bin/systemctl restart rsyslog.service
2023/05/11 11:50:15 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2023/05/11 11:51:37 CLSRSC-672: Post-patch steps for patching GI home successfully completed.

On linux looking at the currently running processes will show the GI related processes running out of new GI home.

 9397 ?        Ssl    0:06 /opt/app/oracle/product/19.19.0/grid/bin/ohasd.bin reboot
 9685 ?        Ssl    0:06 /opt/app/oracle/product/19.19.0/grid/bin/oraagent.bin
 9713 ?        Ssl    0:02 /opt/app/oracle/product/19.19.0/grid/bin/evmd.bin
 9716 ?        Ssl    0:00 /opt/app/oracle/product/19.19.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
 9733 ?        Ss     0:00 /opt/app/oracle/product/19.19.0/grid/opmn/bin/ons -d
 9789 ?        Ssl    0:02 /opt/app/oracle/product/19.19.0/grid/bin/evmlogger.bin -o /opt/app/oracle/product/19.19.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /opt/app/oracle/product/19.19.0/grid/log/[HOSTNAME]/evmd/evmlogger.log
 9805 ?        Ssl    0:02 /opt/app/oracle/product/19.19.0/grid/bin/cssdagent
 9845 ?        Ssl    0:02 /opt/app/oracle/product/19.19.0/grid/bin/ocssd.bin
 9929 ?        Sl     0:00 /opt/app/oracle/product/19.19.0/grid/opmn/bin/ons -d

Resources will be online as before

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-2-77
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-2-77
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-2-77
ora.asm                   ora.asm.type              ONLINE             ONLINE             ip-172-31-2-77
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-2-77
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-2-77
ora.ons                   ora.ons.type              ONLINE             ONLINE             ip-172-31-2-77
ora.testcdb.db            ora.database.type         ONLINE             ONLINE             ip-172-31-2-77
ora.testcdb.dbxrw.svc     ora.service.type          ONLINE             ONLINE             ip-172-31-2-77
ora.testcdb.testsrv.svc   ora.service.type          ONLINE             ONLINE             ip-172-31-2-77

5. Update the inventory by setting CRS=True for new GI home and False for old GI home.

$ /opt/app/oracle/product/19.19.0/grid/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/oracle/product/19.19.0/grid CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 599 MB    Passed
The inventory pointer is located at /etc/oraInst.loc

$ /opt/app/oracle/product/19.x.0/grid/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/oracle/product/19.x.0/grid CRS=FALSE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 599 MB    Passed
The inventory pointer is located at /etc/oraInst.loc

If happy with the results then the old GI home could be deinstalled.



If for whatever reason need to rollback to old GI home after successfully moving to new GI home, then use the same pre and post patch steps using the old GI home.

# /opt/app/oracle/product/19.x.0/grid/crs/install/roothas.sh -prepatch -dstcrshome /opt/app/oracle/product/19.x.0/grid
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/hapatch_2023-05-11_10-45-53AM.log
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/hapatch_2023-05-11_10-45-54AM.log
2023/05/11 10:45:55 CLSRSC-347: Successfully unlock /opt/app/oracle/product/19.x.0/grid
2023/05/11 10:45:55 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.

# /opt/app/oracle/product/19.x.0/grid/crs/install/roothas.sh -postpatch -dstcrshome /opt/app/oracle/product/19.x.0/grid
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/hapatch_2023-05-11_10-46-08AM.log
2023/05/11 10:46:21 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2023/05/11 10:47:47 CLSRSC-672: Post-patch steps for patching GI home successfully completed.

The listener config shows new GI home.

srvctl config listener
Name: LISTENER
Type: Database Listener
Home: /opt/app/oracle/product/19.19.0/grid
End points: TCP:1521
Listener is enabled.

However, if the ASM SPfile is on a FS instead of on ASM them this location is not updated. File must be moved manually and spfile location need to be updated afterwards.

srvctl config asm
ASM home: <CRS home>
Password file:
Backup of Password file:
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: /dev/oracleasm/*

Related Posts
Out of Place (OOP) Patching of Oracle Restart 21c