This post looks at the prerequisite work needed before the upgrade could begin. It is not an extensive look at all the pre-req steps. For complete list of pre-req steps refer the cloud control upgrade guide. Also useful is the MOS doc 2761728.1
One of the first thing to be done as part of pre-req work is to apply the latest patches for both repository database and OMS and related components. The repository database had RU 19.13 applied while OMS was patched with RU 12 (13.4.0.12). The weblogic server had the patch 12.2.1.3.210630. The Oracle documentatin states that if for EM 13.4 RU12 need to be upgraded to 13.5 RU 1 or above. With this in mind download the latest RU for 13.5. For this post the OMS patch downloaded was 13.5 RU 3 (13.5.0.3). This will be applied to the 13.5 binaries during the upgrade. As a pre-req it is enough to download it.
Another patch that was applied to repository database was the patch 29309073. This could be applied after the upgrade as well. Applying it before the upgrade could help in preventing issues with regard to table fragmentation mentioned in 2796903.1, 2169325.1 and 2782869.1.
The full patch list for OMS and repos DB is given below.OMS
opatch lspatches 32720750;Oracle Enterprise Manager for Exadata 13c Release 4 Plug-in Update 12 (13.4.1.12) for Oracle Management Service 32720701;Oracle Enterprise Manager for Fusion Middleware 13c Release 4 Plug-in Update 12 (13.4.1.12) for Oracle Management Service 32720663;Oracle Enterprise Manager for Oracle Database 13c Release 4 Plug-in Update 12 (13.4.1.12) for Oracle Management Service 32720634;Oracle Enterprise Manager 13c Release 4 Platform Update 12 (13.4.0.12) for Oracle Management Service 33115114;OBI BUNDLE PATCH 12.2.1.3.210714 32997257;ADF BUNDLE PATCH 12.2.1.3.210614 1221315;Bundle patch for Oracle Coherence Version 12.2.1.3.15 33064699;WLS PATCH SET UPDATE 12.2.1.3.210630 32436258;Oracle Enterprise Manager for Systems Infrastructure 13c Release 4 Plug-in Update 11 (13.4.1.11) for Oracle Management Service 32397127;OPSS Bundle Patch 12.2.1.3.210420 32668721;OHS (NATIVE) BUNDLE PATCH 12.2.1.3.210323 31971994;OSS BUNDLE PATCH 12.2.1.3.210420 31544340;One-off 31812958;Oracle Enterprise Manager for Cloud Framework 13c Release 4 Plug-in Update 6 (13.4.1.6) for Oracle Management Service 26933408;One-off 31101362; 30510407;One-off 30465861;One-off 30387597;WebCenter Core Bundle Patch 12.2.1.3.0(ID:191002.2131.S) 30152128;One-off 29878681;One-off 28659321;One-off 27570450;One-off 26626168;One-off 26045997;One-off 25977056;One-off 25423296;One-offRepos DB
opatch lspatches 29309073;AQ-CQ PERFORMANCE DEGRADATION IS OBSERVED IN CLASSIC QUEUES DEQUEUE 33192793;Database Release Update : 19.13.0.0.211019 (33192793) 33208123;OCW RELEASE UPDATE 19.13.0.0.0 (33208123)The 13.5 OUI also allows to download necessary patches before or during the upgrade. For this there must be a internet connection from the EMCC server to oracle. If the EMCC server doesn't allow internet connection then patches could be download by running the installer from a server that has internet connection and then move the downloaded patches to server where EMCC is hosted. More on this is available in advance instalaltion guide. To run the installer just for patch download run the following command.
./em13500_linux64.bin DOWNLOAD_UPDATES=trueThis open the OUI with only the software updates option. However, during the testing this feature did not download any patches and stated no updates found.
Pressing the close button says updates are downloaded.
No patches were available on the given location but two xml files patch-metadata.xml and PatchSearch.xml. Only mention of a patch in patchsearch.xml is patch 32648684 which is listed as a "Place holder bug for autoupdate for EM13.5". So it appears this method is either having an issue detecting RU for 13.5 or is not for downloading RU at all.
Below are some pre-req tasks to look out for.
Disable the password verification function on the default profile. If the repository database reside in a PDB then change the default profile on the PDB.
alter profile default limit password_verify_function null;Temporarily disabling the gather stats job. It will be automatically re-enabled at the end of this process.
exec DBMS_AUTO_TASK_ADMIN.DISABLE(client_name => 'auto optimizer stats collection',operation => NULL,window_name => NULL);
Upgrade requires the emkey moved back into the repository. This may have been moved out of repository to enhance security. To check current status use the below command.
emctl status emkey Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : The EMKey is configured properly."The EMKey is configured properly." means that emkey is not in the repository. To move it back to repository use copy_to_repos option.
emctl config emkey -copy_to_repos Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure. After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".Check the status again.
emctl status emkey Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : The EMKey is configured properly, but is not secure. Secure the EMKey by running "emctl config emkey -remove_from_repos"."The EMKey is configured properly, but is not secure" indicates that emkey is residing in the repository.
Grant execute privilege on DBMS_RANDOM to dbsnmp. Compliance standard such as CIS require grant of execute on DBMS_RANDOM to be revoked from public. To avoid possible issues caused by this revoke explicitly grant execute privilege on dbms_random to dbsnmp. As it was not clear if the use of dbms_random was limited to dbsnmp inside the repos db (in this case it is inside a pdb) or generally to dbsnmp (as it is a common user in the CDB) the grant was done container-wide.
grant execute on DBMS_RANDOM to DBSNMP container=all;The 13.5 install and upgrade require certain hidden parameters reset. However, these parameters are expected to be set for 13.4. Therefore, postpone the resetting of these init parameters until just before the run of ConfigureGC.sh. The init parameters could be reset with
alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*'; alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*'; alter system reset "_px_adaptive_dist_method" scope=both sid='*'; alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*'; alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*'; alter system reset "_optimizer_use_feedback" scope=both sid='*'; alter system reset "_optimizer_gather_feedback" scope=both sid='*'; alter system reset "_optimizer_performance_feedback" scope=both sid='*';The 13.5 upgrade also requires job_queue_processes to be set to 0. However, this is need to have a non-zero value for functioning of 13.4. So postpone setting this until before the run of ConfigureGC.sh. The init parameter could be set with
alter system set job_queue_processes=0 scope=both;Disable any logoff triggers. In this system only the following trigger was present.
SELECT trigger_name,owner,TRIGGERING_EVENT FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED'; TRIGGER_NAME OWNER TRIGGERING_EVENT -------------------- -------------------- -------------------- GSMLOGOFF GSMADMIN_INTERNAL LOGOFFDisable it with
SQL> alter trigger GSMADMIN_INTERNAL.GSMLOGOFF disable; Trigger altered.Enable audit for DELETE_TARGET operations. The list of all operations could be found with below command. This is useful in identifying the operation's name.
emcli show_operations_listThe list of enabled audit operations could be found with
emcli login -user=asanga emcli show_audit_settings -view="DETAIL" | grep ENABLED APPLY_UPDATE : ENABLED CHANGE_REPOS_PWD : ENABLED CHANGE_VIEW_USER_PWD : ENABLED COPY_KEY_TO_REPOS : ENABLED LOGIN : ENABLED LOGOUT : ENABLED REMOVE_KEY_FROM_REPOS : ENABLED REMOVE_UPDATE : ENABLED SECURITY_AUTH_CONFIG : ENABLED SECURITY_COMM_CREATE_CA : ENABLED SECURITY_COMM_SECURE_CONSOLE : ENABLED SECURITY_COMM_SECURE_LOCK : ENABLED SECURITY_COMM_SECURE_OMS : ENABLED SECURITY_COMM_SECURE_WLS : ENABLEDTo enable auditing (refer 1499609.1 for more) for DELETE_TARGET with default options use
emcli update_audit_settings -operations_to_enable="DELETE_TARGET" Successfully updated the audit settings. emcli show_audit_settings -view="DETAIL" | grep ENABLED APPLY_UPDATE : ENABLED CHANGE_REPOS_PWD : ENABLED CHANGE_VIEW_USER_PWD : ENABLED COPY_KEY_TO_REPOS : ENABLED DELETE_TARGET : ENABLED LOGIN : ENABLED LOGOUT : ENABLED REMOVE_KEY_FROM_REPOS : ENABLED REMOVE_UPDATE : ENABLED SECURITY_AUTH_CONFIG : ENABLED SECURITY_COMM_CREATE_CA : ENABLED SECURITY_COMM_SECURE_CONSOLE : ENABLED SECURITY_COMM_SECURE_LOCK : ENABLED SECURITY_COMM_SECURE_OMS : ENABLED SECURITY_COMM_SECURE_WLS : ENABLED
One of the easiest way to check the relavent pre-req is to run the installer with EMPREREQ_KIT=true option. For this a respone file is needed. A response file template is given with the name emprereqkit_upgrade.rsp. To extract the response files use the following command ( or unzip the first installer file).
./em13500_linux64.bin -getResponseFileTemplates -outputLoc /opt/xxx/installs/13.5/rsp Launcher log file is /tmp/OraInstall2022-01-26_11-25-17AM/launcher2022-01-26_11-25-17AM.log. Extracting the installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Done Copying response file template(s) to /opt/xxx/installs/13.5/rsp Copying response file template softwareOnlyWithPlugins_install.rsp Copying response file template emprereqkit_upgrade.rsp Copying response file template upgrade.rsp Copying response file template new_install.rsp Copying response file template emprereqkit_install.rsp Copying response file template softwareOnlyWithPlugins_upgrade.rsp Finished copying response file template(s) The log(s) can be found here: /tmp/OraInstall2022-01-26_11-25-17AM.Use the emprereqkit_upgrade.rsp to check upgrade pre-req work. Populate the response file with the relavent values. Below is a sample
cat emprereqkit_upgrade.rsp #################################################################### ## ## ## Copyright (c) 2015 Oracle. All rights reserved. ## ## ## ## Specify values for the variables listed below to customize ## ## your installation. ## ## ## ## Each variable is associated with a comment. The comment ## ## identifies the variable type. ## ## ## ## Specify values in the following format: ## ## ## ## Type Example ## ## ---- ------- ## ## string Sample Value ## ## boolean TRUE or FALSE ## ## number 1000 ## ## ## #################################################################### #################################################################### #------------------------------------------------------------------# # Name : RESPONSEFILE_VERSION # Type : string # Description : Response file name. #------------------------------------------------------------------# RESPONSEFILE_VERSION=2.2.1.0.0 #------------------------------------------------------------------# # Name : UNIX_GROUP_NAME # Type : string # Description : UNIX group name. # Example : UNIX_GROUP_NAME=dba #------------------------------------------------------------------# UNIX_GROUP_NAME=oinstall #------------------------------------------------------------------# # Name : OLD_BASE_DIR # Type : string # Description : Old middleware Home that you want to upgrade.This is applicable when ONE_SYSTEM=true. #------------------------------------------------------------------# OLD_BASE_DIR=/opt/xxx/app/software/em/middleware #------------------------------------------------------------------# # Name : ONE_SYSTEM # Type : boolean # Description : can be true or false. Setting this variable to true indicates # ONE_SYSTEM.Setting to false indicates "TWO_SYSTEM". #------------------------------------------------------------------# ONE_SYSTEM=true #------------------------------------------------------------------# # Name : SYS_PASSWORD # Type : string # Description : Password of DBA user used to create repository schema. # Example : SYS_PASSWORD=password #------------------------------------------------------------------# SYS_PASSWORD=sys_password #------------------------------------------------------------------# # Name : SYSMAN_PASSWORD # Type : string # Description : Password for sysman user. #------------------------------------------------------------------# SYSMAN_PASSWORD=sysman_password #------------------------------------------------------------------# # Name : b_upgrade # Type : boolean # Description : To Specify whether it is Normal Install or Upgrade. #------------------------------------------------------------------# b_upgrade=true #------------------------------------------------------------------# # Name : EM_INSTALL_TYPE # Type : string # Description : Type of Grid Control install being performed. #------------------------------------------------------------------# EM_INSTALL_TYPE=NOSEEDRun the pre-req check by executing the below command.
./em13500_linux64.bin EMPREREQ_KIT=true -silent -responseFile /opt/xxx/installs/13.5/emprereqkit_upgrade.rspIf any of the pre-req fails then an output similar to below would be given.
./em13500_linux64.bin EMPREREQ_KIT=true -silent -responseFile /opt/xxx/installs/13.5/emprereqkit_upgrade.rsp Launcher log file is /tmp/OraInstall2022-01-05_03-05-20PM/launcher2022-01-05_03-05-20PM.log. Extracting the installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Done Checking swap space: must be greater than 512 MB. Actual 1023 MB Passed Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required) Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2022-01-05_03-05-20PM Running EMPREREQ_KIT. ScratchPathValue :/tmp/OraInstall2022-01-05_03-05-20PM Jan 05, 2022 3:06:14 PM org.apache.sshd.common.io.DefaultIoServiceFactoryFactory getIoServiceProvider INFO: No detected/configured IoServiceFactoryFactory using Nio2ServiceFactoryFactory ERROR: There are some failed prereqs exist , Installer can not proceed further. Refer /tmp/OraInstall2022-01-05_03-05-20PM/prereqResultReport.html file for details. The log(s) can be found here: /tmp/OraInstall2022-01-05_03-05-20PM. ERROR: Installer execution failed (232).In this case the directory /tmp/OraInstall2022-01-05_03-05-20PM will have a html file called prereqResultReport.html that will give summary of all passing and failing pre-req items. The directory /tmp/OraInstall2022-01-05_03-05-20PM/emdbprereqs/LATEST has the file emprereqkit.out which gives a textual output of passing and failing pre-req tasks.
On the otherhand if the above command succeeds then the output would be as below.
./em13500_linux64.bin EMPREREQ_KIT=true -silent -responseFile /opt/xxx/installs/13.5/emprereqkit_upgrade.rsp Launcher log file is /tmp/OraInstall2022-01-05_03-22-42PM/launcher2022-01-05_03-22-42PM.log. Extracting the installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Done Checking swap space: must be greater than 512 MB. Actual 1023 MB Passed Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required) Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2022-01-05_03-22-42PM Running EMPREREQ_KIT. ScratchPathValue :/tmp/OraInstall2022-01-05_03-22-42PM Jan 05, 2022 3:23:33 PM org.apache.sshd.common.io.DefaultIoServiceFactoryFactory getIoServiceProvider INFO: No detected/configured IoServiceFactoryFactory using Nio2ServiceFactoryFactory Unable to copy the logs to the permanent location: The permanent log directory is unknown (null).. The log(s) can be found here: /tmp/OraInstall2022-01-05_03-22-42PM.The "Unable to copy the logs to the permanent location" could be ignored (refer 2790714.1).
Even though this is a 13.4 to 13.5 upgrade the prereqResultReport.html will give the following warning.
EM 13.4 doesn't support MD5 certificate. So there's no way agent communication is happening using this. However, there could be other components that could be using this as shown by the output related to this item
more /tmp/OraInstall2022-01-05_03-57-45PM/md5Target.txt /EMGC_GCDomain/GCDomain/BIP|weblogic_j2eeserver|Both|vm-em-console:7845|10.22.0.5|9825,/EMGC_GCDomain/GCDomain/EMGC_ADMINSERVER|weblogic_j2eeserver|Both|vm-em-console:7845|10.22.0.5|6350,/EMGC_GCDomain/GCDomain/EMGC_OMS1|weblogic_j2eeser ver|Both|vm-em-console:7845|10.22.0.5|9250,EM Management Beacon|oracle_beacon|Agent|vm-em-console:7845||5800 Note:- If MD5 usage is detected for EMGC_ADMINSERVER and EMGC_OMS targets in the default WebLogic domain GCDomain, this can be safely ignored as the 13c Upgrade process will reconfigure the Weblogic certificates with SHA-512 algorithm.
The output says, if the warning is related to EMGC_ADMINSERVER and EMGC_OMS targets then it could be ignored. Since 13.5 no longer supports BIP that too could be ignored.
If all the pre-req work is done (or not done in case of init database parameters mentioned earlier) the prereqResultReport.html will have two failed status on "Check if all adaptive features parameters are unset" (related to hidden parameters) and "Check the job_queue_processes instance parameter value" items. These could be rectified just before the repos is upgraded.
There will be a warning status on "Check MD5 certificate usage" which can be ignored.
Three items with into status. They are "Check if Agent running on Oracle Management Service" (agent could be stopped just before repos upgrade), "BI Publisher is no longer configured with Oracle Enterprise Manager 13c Release 5" and "Based on the Bundle Patch (BP) / Release Update (RU) applied in your OMS home, it is strongly recommended to apply the latest Oracle Enterprise Manager 13c Release 5 Updates as available" (will be done during upgrade).
When the pre-req is at this stage the EM is ready for uprade which will be done using the recommended way which is "Upgrade Software Only with Plug-ins and Configure Later". Steps for this will be shown in the next post.
Related Posts
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 2
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 3
Installing Enterprise Manager Cloud Control 13c (13.4)
Converting EM Repository DB from Non-CDB to PDB
Adding Targets on EM Cloud Control 13c
Installing Enterprise Manager Cloud Control 13c (13.3)
Installing Grid Control 11gR1 and Deploying Agents
Upgrading Grid Control 11g to 12c - 1
Useful MOS docs
13.5: Checklist for Upgrading Enterprise Manager Cloud Control from Version 13.3/13.4 to 13.5 [ID 2761728.1]
13.5: Running EMPREREQ_KIT in Silent Mode Throws Error: Unable to copy the logs to the permanent location: The permanent log directory is unknown (null).. [ID 2790714.1]
How To Manage Cloud Control Audit Data with emcli show_audit_settings and emcli update_audit_settings [ID 1499609.1]
EM13c : Health Check Incidents Reported - AQ Tables and Indexes Fragmentation, is in FAILURE [ID 2796903.1]
EM 12c, 13c : Steps to Remove Fragmentation in the EM_EVENT_BUS and EM_NOTIFY_Q AQ Tables Used by Notification Framework [ID 2169325.1]
EM 13.5: EM Console Health Overview Page shows an Alert on "Fragmentation found in Tables: EM_LOADERJOB_Q_TBL" [ID 2782869.1]
EM 13c: GCDomain Coherence Targets Are Showing Down in the OEM Console [ID 2657900.1]
Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) [ID 2760230.2]
