There are three approaches to upgrade as per Enterprise Manager Cloud Control Upgrade Guide (direct quote)
"1-system upgrade, understandably, must always be done on the same platform as the earlier release of the Enterprise Manager system because you are upgrading on the same host.
2-system upgrade is possible on the same platform as the earlier release of the Enterprise Manager system, and from Linux (32-bit) to Linux (64-bit) (or vice versa) because you are upgrading on a remote host.
1-system upgrade on a different host is possible on the same platform as the earlier release of the Enterprise Manager system, and from Linux (32-bit) to Linux (64-bit) (or vice versa) because you are upgrading on a remote host, much like 2-system upgrade approach.".
Comparison of the three approaches.
This blog uses 1-system upgrade on the same host path. The upgrade consists of four main steps. Installing the pre-upgrade patch which allows the agent upgrade. Upgrade of agents and then finally grid control upgrade and post-upgrade steps. Oracle documentation provides a detail breakdown of these steps.
There are few things to know about the upgrade.
Upgrades do not upgrade the database, if the existing database is certified for grid control see 412431.1 then it shouldn't be an issue. For this DB is 11.1.0.7.0.
12c Agents communicate only with 12c OMS so agents are upgrade before OMS and time between agent upgrade and OMS upgrade there won't be any communication between agent and OMS and there won't be any monitoring.
Any agent could be upgrades as long as 12c management software is available.
Do not upgrade EM cloud control on a NFS mounted drive.
EM cloud control will install jdk 1.6_24 and Oracle WebLogic Server 11g Release 1 (10.3.5) if these are not present. For this existing web logic version is 10.3.2
and jdk 1.6_30
Earlier agents ports are carried forward with the upgrade.
Get the pre-upgrade patch and apply it to OMS.
Metalink note How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? [ID 1358092.1] shows the latest patches available for OMS and agents for both unix and windows. At the time of this blog post the latest patch was 13248190 for OMS and 9346243 for Agent. So those were applied. Database had PSU 11.1.0.7.5 applied.
Following metalink notes may help with PSU patch applying issues.
Troubleshooting the Improved Patching Solution for Grid Control 11.1GC Management Server [ID 1326515.1]
Grid Control 11g: post_install script for PSU 4 Patch 12423703 fails with error for "MGMT_SET_JOB_Q_PROC" [ID 1362945.1]
Grid Control 11g: Post Install Script For PSU Patch 12423703 gives Error: "ORA-20110:Unexpected error executing MGMT_SET_JOB_Q_PROC" [ID 1359157.1]
Applying Post Install Scripts For Patch Getting Error: "ORA-00068: invalid value 0 for parameter job_queue_processes" [ID 1375352.1]
Once the PSU patches applied it's time to apply the pre-upgrade console patch 13503844.
13503844]$ /opt/app/oracle/Middleware/oms11g/OPatch/opatch apply
Oracle Interim Patch Installer version 11.1.0.9.4
Copyright (c) 2011, Oracle Corporation. All rights reserved.
Oracle Home : /opt/app/oracle/Middleware/oms11g
Central Inventory : /opt/app/oraInventory
from : /opt/app/oracle/Middleware/oms11g/oraInst.loc
OPatch version : 11.1.0.9.4
OUI version : 11.1.0.8.0
Log file location : /opt/app/oracle/Middleware/oms11g/cfgtoollogs/opatch/13503844_Feb_09_2012_10_56_45/apply2012-02-09_10-56-44AM_1.log
OPatch detects the Middleware Home as "/opt/app/oracle/Middleware"
OPatch will do the following:
[Oracle Home discovery] : Configure and Validate Oracle Home info.
[Prerequisite for apply] : Invoke prerequisites to see if patch can be applied.
[Patch conflict detection for apply patch] : Check if any conflict with already installed patches in Oracle Home.
Applying interim patch '13503844' to OH '/opt/app/oracle/Middleware/oms11g'
Verifying environment and performing prerequisite checks...
To continue, OPatch will do the following:
[Back up for restore] : Back up 68 files for Oracle Home restore in case of patching error.
[Back up for rollback] : Back up 39 files for future patch rollback.
[Patch Oracle Home for apply] : Copy 39 file(s) to Oracle Home;
Archive 4 jar file(s) with 254 class file(s).
[Update inventory for apply] : Modify the inventory to record patch [13503844].
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Backing up files...
Patching component oracle.sysman.oms.core, 11.1.0.1.0...
Verifying the update...
Patch 13503844 successfully applied
Log file location: /opt/app/oracle/Middleware/oms11g/cfgtoollogs/opatch/13503844_Feb_09_2012_10_56_45/apply2012-02-09_10-56-44AM_1.log
OPatch succeeded.
13503844]$ $ORACLE_HOME/bin/rcuJDBCEngine sys/gridb@192.168.0.76:1521:gridb JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/puc_dblink_pkgdef.sql
Completed SQL script execution normally.
1 scripts were processed
13503844]$ $ORACLE_HOME/bin/rcuJDBCEngine sys/gridb@192.168.0.76:1521:gridb JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/puc_dblink_pkgbody.sql
Completed SQL script execution normally.
1 scripts were processed
13503844]$ $ORACLE_HOME/bin/rcuJDBCEngine sysman/gridb321@192.168.0.76:1521:gridb JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/pre_upg_console.sql
l_target_guids.COUNT : 4
oracle.sysman.beacon SMPADDON 11.1.0.1.0 Oracle Beacon Plugin
oracle.sysman.emrep SMPADDON 11.1.0.1.0 Oracle Management And Repository Plugin
oracle.sysman.oh SMPADDON 11.1.0.1.0 Oracle OracleHome
oracle.sysman.csa SMPADDON 11.1.0.1.0 Oracle Client System Analyzer
oracle.sysman.db SMPADDON 11.1.0.1.0 Oracle Database Plugin
oracle.sysman.emas SMPADDON 11.1.0.1.0 Oracle Fusion Middleware Plugin
oracle.sysman.empa ADDON 11.1.0.1.0 Oracle Siebel Plugin
oracle.sysman.vt ADDON 11.1.0.1.0 Oracle Virtualization Plugin
oracle.fmw.bhv ADDON 11.1.0.1.0 Oracle Beehive Plugin
Completed SQL script execution normally.
24 scripts were processedWhen the patch is applied it add a link in the deployment tab page under upgrade topic. Below is a screenshot of this page before pre-upgrade patch is applied.
Download agent core plugin and other needed plugins and copy these files into a directory in the host where grid control is running.
Below is the list of downloaded plugins.
-rw-r--r-- 1 oracle oinstall 495016 Feb 9 11:58 12.1.0.1.0_oracle.sysman.oh_2000_0.opar -rw-r--r-- 1 oracle oinstall 499849 Feb 9 11:58 12.1.0.1.0_oracle.sysman.csa_2000_0.opar -rw-r--r-- 1 oracle oinstall 297231 Feb 9 11:58 12.1.0.1.0_oracle.sysman.emrep_2000_0.opar -rw-r--r-- 1 oracle oinstall 3925371 Feb 9 11:58 12.1.0.1.0_oracle.em.soee_2000_0.opar -rw-r--r-- 1 oracle oinstall 230699506 Feb 9 11:59 12.1.0.1.0_AgentCore_226.zip -rw-r--r-- 1 oracle oinstall 223979520 Feb 9 11:59 12.1.0.1.0_AgentCore_46.zip -rw-r--r-- 1 oracle oinstall 621251038 Feb 9 11:59 12.1.0.1.0_oracle.sysman.db_2000_0.opar -rw-r--r-- 1 oracle oinstall 106452968 Feb 9 12:08 12.1.0.1.0_oracle.sysman.beacon_2000_0.opar -rw-r--r-- 1 oracle oinstall 440206388 Feb 9 12:17 12.1.0.1.0_oracle.sysman.emas_2000_0.oparSelect 1-system upgrade type and then click mange software link in the upgrade console page. This will show the agent upgradeability. without the above plugin being discovered agents will be shown as unable to upgrade. Below screenshots show the current state of agent upgradeability as no plugin or agent software is validated.
Give the directory location of the software and validate.
When the plugins and agent software is validate all the agents become upgradable. Resolve any issues, missing plugins and re-validate before continuing.
plugin_bin]$ ls -lrt drwxr----- 3 oracle oinstall 4096 Feb 9 12:18 oracle.sysman.oh drwxr----- 3 oracle oinstall 4096 Feb 9 12:18 oracle.sysman.emrep drwxr----- 3 oracle oinstall 4096 Feb 9 12:18 oracle.sysman.emas drwxr----- 3 oracle oinstall 4096 Feb 9 12:18 oracle.em.soee drwxr----- 3 oracle oinstall 4096 Feb 9 12:18 oracle.sysman.beacon drwxr----- 3 oracle oinstall 4096 Feb 9 12:18 oracle.sysman.db drwxr----- 3 oracle oinstall 4096 Feb 9 12:19 oracle.sysman.csaClicking on the upgradable link next to each green pie chart will show the upgradeability of each agent and each target.
Once all agents are upgradeable click on "Agent Upgrade Status" link which is under "Other Links" in upgrade console page.At this stage all the status should be not started.
Click deploy and configure agents button. Multiple agents could be selected to be deployed and configured but this require host and agent credentials if these are not set then oracle and root credentials must be same for all hosts. Here agents are deployed and configured one at a time. Use credentials, selecting sudo gave error "sudo is not present".
Select the host deploy and configure the new agent.
Clicking the deploy and configure button open up the page below which allows to specify the new directory for agent host credentials.
Click continue and give root user credentials.
When the submit button is clicked it will start the deploy and configure job.
As said earlier it is also possible to deploy and configure on multiple hosts. This is useful in RAC environments.
When the job has finished the status will be shown. If job fails inspect the logs which could be accessed through the links given as steps in the same page.
Once the agent deployment is successful run agent readiness check.
At the end of agent readiness check configuration and deployment status will be success.
At the end agent readiness check Health Check Status column has "report pending verification". This requires manually verifying it is alright to proceed. Click on view and verify health check report button and inspect the report and confirm if it is error free.
After this agent is ready for the switch.
This where the down time starts. When switch agent is started the current 11g agent will be stopped and disabled. The 12c new agent will be started but cannot report to 11g OMS until it's upgrade. Therefore until the OMS is upgraded there will be no monitoring. To keep this unmonitored time to a minimum carry out the steps before "switch agent" all the agents. Considering the agents listed in this blog agent on host hpc1 will not be switched until hpc4 and rac1 and rac2 all have completed their agent deployments and health checks.
Agent switch could be done one agent a time or multiple agents at the same time. Below are the screenshots from switching agents on all the RAC nodes at the same time.
Agent upgrade status after all the agents are switched.
When all the agents are switched to 12c agents and monitoring has effectively halted until OMS is upgraded. Status of the new agent states its up and ready but still unable to communicate with OMS.
./emctl status agent Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.1.0 OMS Version : (unknown) Protocol Version : 12.1.0.1.0 Agent Home : /opt/app/oracle/grid_agent/agent12g/agent_inst Agent Binaries : /opt/app/oracle/grid_agent/agent12g/core/12.1.0.1.0 Agent Process ID : 4072 Parent Process ID : 3935 Agent URL : https://rac1. .net:3872/emd/main Repository URL : https://hpc4. .net:4900/empbs/upload Started at : 2012-02-09 16:26:48 Started by user : oracle Last Reload : (none) Last successful upload : (none) Last attempted upload : (none) Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload : 82 Size of XML files pending upload(MB) : 0.54 Available disk space on upload filesystem : 13.59% Collection Status : Collections enabled Last attempted heartbeat to OMS : 2012-02-09 16:32:58 Last successful heartbeat to OMS : (none) --------------------------------------------------------------- Agent is Running and ReadyTo keep the un-monitored time to a minimum begin the OMS upgrade soon after the agent switch.
Part 2 shows the OMS upgrade.
Useful metalink notes
Upgrade Advisor: Enterprise Manager 1-System Upgrade from 11.1.0.1 to 12.1.0.1 [ID 328.1]
12c: Upgrading the 11.1.0.1 Enterprise Manager on Linux x86_64 to 12.1.0.1 with the 1-System Method [ID 1364931.1]
How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? [ID 1358092.1]
