Wednesday, February 23, 2022

Oracle Restart Install Failures on RHEL8

19.6 or 19.7 DB software could be installed in either RHEL8 or OEL8. There is a dedicated MOS doc on this (2668780.1). However, there's no such document exists for Oracle Restart. Following GI and DB installation guide for 19c would be helpful in getting Oracle restart installed. Latest edition of those install guides list RHEL8/OL8 as valid operating systems.
One of the packages listed on the MOS doc is libnsl12-devel. To install this via dnf enable codeready-builder-for-rhel-8-rhui-rpms in redhat-rhui.repo (when installing on RHEL8 on AWS). To get installer going the distribution is assumed to be OL7. This can result in a false pre-req failure.
export CV_ASSUME_DISTID=OL7
/opt/app/oracle/product/19.x.0/grid/runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: ip-172-31-2-172:/usr,ip-172-31-2-172:/var,ip-172-31-2-172:/etc,ip-172-31-2-172:/sbin,ip-172-31-2-172:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...FAILED (PRVF-7532)
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying Swap Size ...FAILED
ip-172-31-2-172: PRVF-7573 : Sufficient swap size is not available on node
                 "ip-172-31-2-172" [Required = 15.459GB (1.6209956E7KB) ; Found
                 = 1.25GB (1310716.0KB)]

Verifying Package: compat-libcap1-1.10 ...FAILED
ip-172-31-2-172: PRVF-7532 : Package "compat-libcap1" is missing on node
                 "ip-172-31-2-172"
This failure on package compat-libcap1 is safe to ignore. Refer 2801988.1 for more.



Another failure happens when root.sh is run. Following line could be seen on the log
Failure in execution (rc=-1, 0, No such file or directory)
To fix this install policycoreutils-python-utils package. This is not listed as one of the required rpms nor check when cluvfy is run. Refer 2825802.1 for more.
Beside these failures there's no other additonal steps to look out for. Oracle restart steup can continue same as RHEL7/OL7.

Related Posts
Installing Oracle Restart Without ASM
Oracle Extended Cluster Setup on 19c
Installing 19c (19.3) RAC on RHEL 7 Using Response File
Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Sunday, February 6, 2022

Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 3

This is the third post on upgrading EMCC from 13.4 to 13.5. Previous posts showed how to complete the pre-req work and upgrade the EM. This post shows how to upgrade the central agent to 13.5.
Central agent could be upgraded either using the EM console or emcli. This post shows the steps for upgrading using EM Console.
To upgrade the central agent go to Setup -> Manage Cloud Control -> Upgrade Agents
This will open the agent upgrade console. Agent Upgrade Tasks tab will be selected. Under Agents for Upgrade click the Add button. This will open upgradeable agents box and select the current 13.4 central agent. Once selected the agent details will be populated as shown below. It shows current version, version it will be upgraded to and the agent home.

Under the Choose credentails select "Preferred Privileged Credentials" (selected by default) even if no perferred credentails are available. This is not an issue as explained in the upgrade document "You can upgrade a Management Agent using the Agent Upgrade Console or EM CLI even when you do not have preferred privileged credentials or non-privileged credentials set, or are not aware of the Management Agent credentials. Privileged credentials are only required to run the root.sh script post-upgrade. If you upgrade a Management Agent as a user who does not have root privileges, or you upgrade a Management Agent without having preferred privileged credentials, a warning appears. You can ignore this warning during the upgrade. Later, you can log in to the Management Agent host as the root user, and run the $<AGENT_BASE_DIR>/agent_13.5.0.0.0/root.sh script".
This is also mentioned when the submit button is clicked. Click OK to continue with the upgrade.

The upgrade job could be monitored under job activity.

Wait for upgrade job to complete successfully.

Agent upgrade results could also be viewd by clicking the job name available under "Agent Upgrade Results" page.

The agent will have a new home under agent base directory.
 pwd
/opt/xxx/app/software/em/agent

ls
agent_13.4.0.0.0  agent_13.5.0.0.0  agentimage.properties  agent_inst  agentInstall.rsp  backup_agtup
Run the root script.
# ./root.sh

Finished product-specific root actions.
/etc exist
Agent version is now upgraded to 13.5
emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.5.0.0.0
OMS Version            : 13.5.0.0.0
Protocol Version       : 12.1.0.1.0


However, it is still running base release and need to be patched. The patches installed in the agent at this stage are
opatch lspatches
32574981;
32313251;
32302527;

OPatch succeeded.
To patch agents in 13.5 instead of opatch the agentpatcher must be used (see 2810322.1). Download agentpatcher and install it in the new 13.5 agent home. Check the agent patcher version is the correct version for the agent RU.
agentpatcher version
AgentPatcher Version: 13.9.5.1.0
OPlan Version: 12.2.0.1.16
OsysModel build: Tue Apr 28 18:16:31 PDT 2020
Install the 13.5 agent RU3 patch (33588930) using agent patcher.
agentpatcher apply -analyze

which emctl
/opt/xxx/app/software/em/agent/agent_13.5.0.0.0/bin/emctl

emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

agentpatcher apply
AgentPatcher Automation Tool
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


AgentPatcher version : 13.9.5.1.0
OUI version        : 13.9.4.0.0
Running from       : /opt/xxx/app/software/em/agent/agent_13.5.0.0.0
Log file location  : /opt/xxx/app/software/em/agent/agent_13.5.0.0.0/cfgtoollogs/agentpatcher/opatch2022-01-17_15-25-25PM_1.log

AgentPatcher log file: /opt/xxx/app/software/em/agent/agent_13.5.0.0.0/cfgtoollogs/agentpatcher/33588930/agentpatcher_2022-01-17_15-25-26PM_deploy.log



Prereq "checkComponents" for patch 33586871 passed.
WARNING: Could not apply the patch "33587645" because the "oracle.sysman.am.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "32968787" because the "oracle.sysman.bda.discovery.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "32921727" because the "oracle.sysman.empa.discovery.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33587537 passed.
WARNING: Could not apply the patch "33587676" because the "oracle.sysman.bda.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33586858 passed.

Prereq "checkComponents" for patch 33602498 passed.

Prereq "checkComponents" for patch 33455336 passed.

Prereq "checkComponents" for patch 33586844 passed.
WARNING: Could not apply the patch "33455281" because the "oracle.sysman.empa.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33587629" because the "oracle.sysman.vt.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33586784 passed.

Prereq "checkComponents" for patch 33587056 passed.

Prereq "checkComponents" for patch 33586861 passed.
WARNING: Could not apply the patch "33587519" because the "oracle.sysman.xa.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33587733" because the "oracle.sysman.vi.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33586851" because the "oracle.sysman.emfa.discovery.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33586864" because the "oracle.sysman.emfa.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Running apply prerequisite checks for sub-patch(es) "33586844,33586861,33455336,33586784,33587056,33587537,33602498,33586871,33586858" and Oracle Home "/opt/xxx/app/software/em/agent/agent_13.5.0.0.0"...
Sub-patch(es) "33586844,33586861,33455336,33586784,33587056,33587537,33602498,33586871,33586858" are successfully analyzed for Oracle Home "/opt/xxx/app/software/em/agent/agent_13.5.0.0.0"

To continue, AgentPatcher will do the following:
[Patch and deploy artifacts]   :


Do you want to proceed? [y|n]
y
User Responded with: Y

Applying sub-patch(es) "33455336,33586784,33586844,33586858,33586861,33586871,33587056,33587537,33602498"
Please monitor log file: /opt/xxx/app/software/em/agent/agent_13.5.0.0.0/cfgtoollogs/opatch/opatch2022-01-17_15-25-27PM_1.log


Complete Summary
================


All log file names referenced below can be accessed from the directory "/opt/xxx/app/software/em/agent/agent_13.5.0.0.0/cfgtoollogs/agentpatcher/2022-01-17_15-25-25PM_SystemPatch_33588930_1"

Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:

                          Featureset                                                                        Sub-patches                                                                                                             Log file
                          ----------                                                                        -----------                                                                                                             --------
  oracle.sysman.top.agent_13.5.0.0.0   33455336,33586784,33586844,33586858,33586861,33586871,33587056,33587537,33602498   33455336,33586784,33586844,33586858,33586861,33586871,33587056,33587537,33602498_opatch2022-01-17_15-25-27PM_1.log


The following sub-patches are incompatible with components installed in the Agent system:
33587645,32968787,32921727,33587676,33455281,33587629,33587519,33587733,33586851,33586864



--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  Could not apply the patch "33587645" because the "oracle.sysman.am.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
2)  Could not apply the patch "32968787" because the "oracle.sysman.bda.discovery.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
3)  Could not apply the patch "32921727" because the "oracle.sysman.empa.discovery.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
4)  Could not apply the patch "33587676" because the "oracle.sysman.bda.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
5)  Could not apply the patch "33455281" because the "oracle.sysman.empa.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
6)  Could not apply the patch "33587629" because the "oracle.sysman.vt.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
7)  Could not apply the patch "33587519" because the "oracle.sysman.xa.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
8)  Could not apply the patch "33587733" because the "oracle.sysman.vi.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
9)  Could not apply the patch "33586851" because the "oracle.sysman.emfa.discovery.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
10)  Could not apply the patch "33586864" because the "oracle.sysman.emfa.agent.plugin with version 13.5.1.0.0" core component of the "Agent" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
--------------------------------------------------------------------------------
AgentPatcher Session completed with warnings.
Log file location: /opt/xxx/app/software/em/agent/agent_13.5.0.0.0/cfgtoollogs/agentpatcher/33588930/agentpatcher_2022-01-17_15-25-26PM_deploy.log

AgentPatcher completed with warnings.

emctl start agent

Patches applied on agent
opatch lspatches
33602498;Oracle Enterprise Manager for Exadata 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Agent (Discovery)
33587537;Oracle Enterprise Manager for Oracle Home 13c Release 5 Plug-in Update 3 (13.5.0.3) for Oracle Management Agent
33587056;Oracle Enterprise Manager for EMREP 13c Release 5 Plug-in Update 3 (13.5.0.3) for Oracle Management Agent
33586871;Oracle Enterprise Manager 13c Release 5 Platform Update 3 (13.5.0.3) for Oracle Management Agent
33586861;Oracle Enterprise Manager for Fusion Middleware 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Agent
33586858;Oracle Enterprise Manager for Oracle Database 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Agent
33586844;Oracle Enterprise Manager for Fusion Middleware 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Agent (Discovery)
33586784;Oracle Enterprise Manager for Oracle Database 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Agent (Discovery)
33455336;Oracle Enterprise Manager for Systems Infrastructure 13c Release 5 Plug-in Update 2 (13.5.1.2) for Oracle Management Agent
32574981;
32313251;
32302527;
As the last step the old 13.4 agent could be removed. This is done as a post agent upgrade task. Click on the cleanup agents and add the old 13.4 agent. It was noticed in multiple upgrades that the "installed version" for the old agent is listed as 13.5 even though correct 13.4 agent home is listed as the oracle home. Ignore this possible bug on the EM and continue with the 13.4 agent removal.

As before the agent removal progress could be monitored under job activity. Once complete view the results in cleanup-agent results page.

The old 13.4 agent directory would have been removed from the agent base location.
 pwd
/opt/xxx/app/software/em/agent

ls
agent_13.5.0.0.0  agentimage.properties  agent_inst  agentInstall.rsp
This concludes the upgrading of Enterprise Manager Cloud Control from 13.4 to 13.5.

Related Posts
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 1
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 2
Installing Enterprise Manager Cloud Control 13c (13.4)
Converting EM Repository DB from Non-CDB to PDB
Adding Targets on EM Cloud Control 13c
Installing Enterprise Manager Cloud Control 13c (13.3)
Installing Grid Control 11gR1 and Deploying Agents
Upgrading Grid Control 11g to 12c - 1

Useful MOS Docs
13.5: How to Upgrade AgentPatcher to the Version 13.9.5.1.0 [ID 2810322.1]

Tuesday, February 1, 2022

Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 2

This is the second post on upgrading EMCC from 13.4 to 13.5 The previous post mentioned pre-req tasks to be carried out before the installation. The approach used in this case consists of two phase. First phase installs the 13.5 binaries and patch them. The existing 13.4 EM could be up and running at this stage. The second phase is the actual configuration of 13.5 EM and upgrade of the schemas in the repository. Prior to the second phase the existing 13.4 EM and agents must be shutdown. Also any pre-req work that was postponed (i.e. database init parameter changes) must be completed. This two phase approach reduces the total downtime needed for upgrading.
It is possible to run the upgrade either in GUI mode or silent mode using the softwareOnlyWithPlugins_upgrade.rsp extracted from the installers. In this case the GUI was chosen. To start the binary installation execute the following
./em13500_linux64.bin
This will open the GUI. Select upgrade existing EM and under it select the option upgrade software only and configure later.

Skip the software update download. Previous post showed how (if it works) patches could be downloaded beforehand.


Ensure no pre-req failures.

Give new location for the middleware.

Give repository DB connection details

Ensure following DB pre-req succeeds.

Two DB pre-req will have warnings. One of them is the use of MD5 which was covered in the previous post and is safe to ignore.

The other warning is related to applying of applyg RU 1 (which is minimum version needed when upgrading from 13.4 RU12). Since this will be done after binary install, this warning too could be ignored.

When the next button is clicked following information will be shown.

Upgrade will take care of some plugin by upgrading or migrating them depending on availability.

Select any additional plugin to deploy.

Review Space and install location.

Click the upgrade button to begin the upgrade.

When the upgrade completes run the allroot.sh.
# /opt/xxx/app/software/em/middleware135/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /opt/xxx/app/software/em/middleware135/root.sh ......
Check /opt/xxx/app/software/em/middleware135/install/root_vm-em-console_2022-01-11_15-17-28.log for the output of root script
Finished execution of  /opt/xxx/app/software/em/middleware135/root.sh ......
Once the allroot.sh is run click the OK button. This will complete the install of binaries and give instructions on the next step which is to configure the setup but do not run the ConfigureGC.sh until the binaries are patched. It is alright to click the close button before the patch apply.




Download the latest omsopatcher (patch 19999993) from Oracle (refer 2809842.1) and install in the new 13.5 middleware home.
export ORACLE_HOME=/opt/xxx/app/software/em/middleware135
export PATH=$ORACLE_HOME/bin:$PATH
export PATH=$ORACLE_HOME/OMSPatcher:$PATH
export PATH=$ORACLE_HOME/OPatch:$PATH

which omspatcher
/opt/xxx/app/software/em/middleware135/OMSPatcher/omspatcher

omspatcher version
OMSPatcher Version: 13.9.4.4.0
OPlan Version: 12.2.0.1.16
OsysModel build: Tue Apr 28 18:16:31 PDT 2020

OMSPatcher succeeded.

unzip p19999993_135000_Generic_oms_patcher.zip -d $ORACLE_HOME/

omspatcher version
OMSPatcher Version: 13.9.5.1.0
OPlan Version: 12.2.0.1.16
OsysModel build: Tue Apr 28 18:16:31 PDT 2020

OMSPatcher succeeded.
Apply the 13.5 RU3 patch to the new 13.5 middleware home (refer 2810169.1). Patch analysis output is shown below
unzip p33588912_135000_Generic_oms_base_13.5.0.3.zip
cd 33588912

omspatcher apply -analyze -bitonly
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.1.0
OUI version        : 13.9.4.0.0
Running from       : /opt/xxx/app/software/em/middleware135
Log file location  : /opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/opatch2022-01-11_15-25-06PM_1.log

OMSPatcher log file: /opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/33588912/omspatcher_2022-01-11_15-25-07PM_analyze.log

WARNING: OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.

Do you want to proceed? [y|n]
y
User Responded with: Y


Prereq "checkComponents" for patch 33424163 passed.

Prereq "checkComponents" for patch 32941631 passed.
WARNING: Could not apply the patch "33424187" because the "oracle.sysman.smf.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424202" because the "oracle.sysman.vt.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "32972412" because the "oracle.sysman.empa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33424205 passed.
WARNING: Could not apply the patch "33424177" because the "oracle.sysman.am.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33424166 passed.

Prereq "checkComponents" for patch 33424169 passed.
WARNING: Could not apply the patch "33424180" because the "oracle.sysman.bda.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424183" because the "oracle.sysman.emct.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424173" because the "oracle.sysman.emfa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424196" because the "oracle.sysman.ssa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33424193 passed.
WARNING: Could not apply the patch "33424208" because the "oracle.sysman.vi.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Running apply prerequisite checks for sub-patch(es) "33424169,32941631,33424166,33424163,33424193,33424205" and Oracle Home "/opt/xxx/app/software/em/middleware135"...
Sub-patch(es) "33424169,32941631,33424166,33424163,33424193,33424205" are successfully analyzed for Oracle Home "/opt/xxx/app/software/em/middleware135"


Complete Summary
================


All log file names referenced below can be accessed from the directory "/opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/2022-01-11_15-25-06PM_SystemPatch_33588912_1"

Prerequisites analysis summary:
-------------------------------

The following sub-patch(es) are applicable:

             Featureset                                             Sub-patches                                                                                  Log file
             ----------                                             -----------                                                                                  --------
  oracle.sysman.top.oms   33424169,32941631,33424166,33424163,33424193,33424205   33424169,32941631,33424166,33424163,33424193,33424205_opatch2022-01-11_15-25-10PM_1.log


The following sub-patches are incompatible with components installed in the OMS system:
33424187,33424202,32972412,33424177,33424180,33424183,33424173,33424196,33424208



--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.
2)  Could not apply the patch "33424187" because the "oracle.sysman.smf.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
3)  Could not apply the patch "33424202" because the "oracle.sysman.vt.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
4)  Could not apply the patch "32972412" because the "oracle.sysman.empa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
5)  Could not apply the patch "33424177" because the "oracle.sysman.am.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
6)  Could not apply the patch "33424180" because the "oracle.sysman.bda.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
7)  Could not apply the patch "33424183" because the "oracle.sysman.emct.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
8)  Could not apply the patch "33424173" because the "oracle.sysman.emfa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
9)  Could not apply the patch "33424196" because the "oracle.sysman.ssa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
10)  Could not apply the patch "33424208" because the "oracle.sysman.vi.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
--------------------------------------------------------------------------------
OMSPatcher Session completed with warnings.
Log file location: /opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/33588912/omspatcher_2022-01-11_15-25-07PM_analyze.log

OMSPatcher completed with warnings.

Patches before the RU apply
opatch lspatches
32458315;ADF BUNDLE PATCH 12.2.1.4.210203
32412974;One-off
31818221;One-off
31808404;OHS (NATIVE) BUNDLE PATCH 12.2.1.4.200826
31708760;One-off
31666198;OPSS Bundle Patch 12.2.1.4.200724
30152128;One-off
26626168;One-off
122146;Bundle patch for Oracle Coherence Version 12.2.1.4.6
32253037;WLS PATCH SET UPDATE 12.2.1.4.201209

OPatch succeeded.
Applyign the RU with bitonly option
omspatcher apply -bitonly
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.1.0
OUI version        : 13.9.4.0.0
Running from       : /opt/xxx/app/software/em/middleware135
Log file location  : /opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/opatch2022-01-11_15-28-18PM_1.log

OMSPatcher log file: /opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/33588912/omspatcher_2022-01-11_15-28-18PM_deploy.log

WARNING: OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.

Do you want to proceed? [y|n]
y
User Responded with: Y


Prereq "checkComponents" for patch 33424163 passed.

Prereq "checkComponents" for patch 32941631 passed.
WARNING: Could not apply the patch "33424187" because the "oracle.sysman.smf.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424202" because the "oracle.sysman.vt.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "32972412" because the "oracle.sysman.empa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33424205 passed.
WARNING: Could not apply the patch "33424177" because the "oracle.sysman.am.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33424166 passed.

Prereq "checkComponents" for patch 33424169 passed.
WARNING: Could not apply the patch "33424180" because the "oracle.sysman.bda.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424183" because the "oracle.sysman.emct.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424173" because the "oracle.sysman.emfa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "33424196" because the "oracle.sysman.ssa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Prereq "checkComponents" for patch 33424193 passed.
WARNING: Could not apply the patch "33424208" because the "oracle.sysman.vi.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.

Running apply prerequisite checks for sub-patch(es) "33424169,32941631,33424166,33424163,33424193,33424205" and Oracle Home "/opt/xxx/app/software/em/middleware135"...
Sub-patch(es) "33424169,32941631,33424166,33424163,33424193,33424205" are successfully analyzed for Oracle Home "/opt/xxx/app/software/em/middleware135"

To continue, OMSPatcher will do the following:
[Patch and deploy artifacts]   :


Do you want to proceed? [y|n]
y
User Responded with: Y

Applying sub-patch(es) "32941631,33424163,33424166,33424169,33424193,33424205"
Please monitor log file: /opt/xxx/app/software/em/middleware135/cfgtoollogs/opatch/opatch2022-01-11_15-28-21PM_1.log


Complete Summary
================


All log file names referenced below can be accessed from the directory "/opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/2022-01-11_15-28-18PM_SystemPatch_33588912_1"

Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:

                        Featureset                                             Sub-patches                                                                                  Log file
                        ----------                                             -----------                                                                                  --------
  oracle.sysman.top.oms_13.5.0.0.0   32941631,33424163,33424166,33424169,33424193,33424205   32941631,33424163,33424166,33424169,33424193,33424205_opatch2022-01-11_15-28-21PM_1.log


The following sub-patches are incompatible with components installed in the OMS system:
33424187,33424202,32972412,33424177,33424180,33424183,33424173,33424196,33424208



--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  OMSPatcher has been invoked with bitonly option but the System patch provided has deployment metadata.
Invocation in bitonly mode will prevent OMSPatcher from deploying artifacts.
2)  Could not apply the patch "33424187" because the "oracle.sysman.smf.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
3)  Could not apply the patch "33424202" because the "oracle.sysman.vt.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
4)  Could not apply the patch "32972412" because the "oracle.sysman.empa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
5)  Could not apply the patch "33424177" because the "oracle.sysman.am.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
6)  Could not apply the patch "33424180" because the "oracle.sysman.bda.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
7)  Could not apply the patch "33424183" because the "oracle.sysman.emct.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
8)  Could not apply the patch "33424173" because the "oracle.sysman.emfa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
9)  Could not apply the patch "33424196" because the "oracle.sysman.ssa.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
10)  Could not apply the patch "33424208" because the "oracle.sysman.vi.oms.plugin with version 13.5.1.0.0" core component of the "OMS" or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
--------------------------------------------------------------------------------
OMSPatcher Session completed with warnings.
Log file location: /opt/xxx/app/software/em/middleware135/cfgtoollogs/omspatcher/33588912/omspatcher_2022-01-11_15-28-18PM_deploy.log

OMSPatcher completed with warnings.

Patches after apply
opatch lspatches
33424205;Oracle Enterprise Manager for Systems Infrastructure 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Service
33424193;Oracle Enterprise Manager for Exadata 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Service
33424169;Oracle Enterprise Manager for Oracle Database 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Service
33424166;Oracle Enterprise Manager for Fusion Middleware 13c Release 5 Plug-in Update 3 (13.5.1.3) for Oracle Management Service
33424163;Oracle Enterprise Manager 13c Release 5 Platform Update 3 (13.5.0.3) for Oracle Management Service
32941631;Oracle Enterprise Manager for Cloud Framework 13c Release 5 Plug-in Update 1 (13.5.1.1) for Oracle Management Service
32458315;ADF BUNDLE PATCH 12.2.1.4.210203
32412974;One-off
31818221;One-off
31808404;OHS (NATIVE) BUNDLE PATCH 12.2.1.4.200826
31708760;One-off
31666198;OPSS Bundle Patch 12.2.1.4.200724
30152128;One-off
26626168;One-off
122146;Bundle patch for Oracle Coherence Version 12.2.1.4.6
32253037;WLS PATCH SET UPDATE 12.2.1.4.201209

OPatch succeeded.
After RU is applied the next step is to run the ConfigureGC.sh. Before running this stop the oms with -all option
emctl stop oms -all
Also all agents connecting to the OMS.
emctl stop agent
Carry out any DB pre-req work postponed. In this case the "Check if all adaptive features parameters are unset" and "Check the active job execution count before a job type upgrade" were postponed. So those will be carried out now. If these parameters were set at PDB level instead of CDB level then make sure reset is run on PDB.
 alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*'; 
 alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*';
 alter system reset "_px_adaptive_dist_method" scope=both sid='*';
 alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*';
 alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*';
 alter system reset "_optimizer_use_feedback" scope=both sid='*';
 alter system reset "_optimizer_gather_feedback" scope=both sid='*';
 alter system reset "_optimizer_performance_feedback" scope=both sid='*';
 
 alter system set job_queue_processes=0 scope=both;
Set the ORACLE_HOME variable to new OMS home and run the ConfigureGC.sh.
export ORACLE_HOME=/opt/xxx/app/software/em/middleware135
$ORACLE_HOME/sysman/install/ConfigureGC.sh
This will open GUI. Select upgrade existing EM system

At this stage the DB modification will happen as such its a good idea to have a backup of the DB.

Ensure all DB pre-req are successful.

The extend weblogic server domain step comes pre-populated with server,port and the weblgoic user names. Enter the password for the user and the base location.

The port configuration will come pre-populated with existing ports used for configuration.

Review and click configure button.

Configuration assitants show the progress of each step

At the end of the configuration the finish page shows the EM URL and port along with admin service information.




With this the EM is upgraded to 13.5. The status command shows the following
emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up

emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Console Server Host        : xxx
HTTP Console Port          : xxx
HTTPS Console Port         : xx
HTTP Upload Port           : xx
HTTPS Upload Port          : xx
EM Instance Home           : /opt/xxx/app/software/em/gc_inst135/em/EMGC_OMS1
OMS Log Directory Location : /opt/xxx/app/software/em/gc_inst135/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 2
Console URL: https://xxx:xxx/em
Upload URL: https://xxx:xxx/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : xxx
Admin Server HTTPS Port: xxx
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: xxx
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
The repost key will also be secured.
emctl status emkey
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.
If the central agent has JDK 1.8u261 applied then it could be started and will start monitoring the 13.5 OMS. Without JDK 1.8u261 the agent will run into issues and won't be able to monitor the OMS until upgraded. In this case the agent had JDK 1.8.0_291 applied so was able to start and monitor OMS before the agent upgrade.
emctl start agent
emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.4.0.0.0
OMS Version            : 13.5.0.0.0
Protocol Version       : 12.1.0.1.0
...
During the configuration the job_queue_processes will be set to 50. This could be seen on the alert log.
pdbname(3):ALTER SYSTEM SET job_queue_processes=50 SCOPE=BOTH SID='cdbname' PDB='pdbname';
If password verification function was set to null set it back to original value.

To remove the old 13.4 middleware home verify no processes are running out of it and detach it from inventory.
ps ax | grep  /middleware/
echo $ORACLE_HOME
/opt/xxx/app/software/em/middleware

$ORACLE_HOME/oui/bin/detachHome.sh -silent ORACLE_HOME=$ORACLE_HOME
Launcher log file is /tmp/OraInstall2022-01-17_03-43-48PM/launcher2022-01-17_03-43-48PM.log.
Checking swap space: must be greater than 500 MB.   Actual 1023 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
detachHome failed.

The log(s) can be found here: /tmp/OraInstall2022-01-17_03-43-48PM.
Though message says detachment fails the inventory.xml says the old home has been rmeoved.
more /opt/xxx/app/oraInventory/ContentsXML/inventory.xml
...
<HOME_LIST>
      <HOME NAME="oms13c1" LOC="/opt/xxx/app/software/em/middleware" TYPE="O" IDX="3" REMOVED="T"/>
<HOME_LIST>
Oracle documentation also states to detach oracle_common directory
$ORACLE_HOME/oui/bin/detachHome.sh -silent ORACLE_HOME=$ORACLE_HOME/oracle_common
but as there's no such entry in the inventory this fails and could be ignored.
Once detached from oracle inventory the old middleware home and instance base location could be removed.
ls
agent  gc_inst  gc_inst135  middleware  middleware135  swlib

rm -rf middleware gc_inst

ls
agent  gc_inst135  middleware135  swlib
The next post will show how to upgrade the central agent to 13.5.

Related Posts
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 1
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 3
Installing Enterprise Manager Cloud Control 13c (13.4)
Converting EM Repository DB from Non-CDB to PDB
Adding Targets on EM Cloud Control 13c
Installing Enterprise Manager Cloud Control 13c (13.3)
Installing Grid Control 11gR1 and Deploying Agents
Upgrading Grid Control 11g to 12c - 1

Useful MOS Docs
13.5: How To Upgrade Enterprise Manager 13.5 Cloud Control OMSPatcher Utility to Version 13.9.5.1.0 [ID 2809842.1]
13.5: How to Apply Release Update on the OMS During the Install/Upgrade [ID 2810169.1]
EM 13c: GCDomain Coherence Targets Are Showing Down in the OEM Console [ID 2657900.1]
How to Refresh or Remove the Obsolete Cache Members in the EM Console for the Coherence Cluster? [ID 2765636.1]