Saturday, February 11, 2012

Upgrading Grid Control 11g to 12c - 1

This blog upgrades the earlier installed Grid Control 11g.

There are three approaches to upgrade as per Enterprise Manager Cloud Control Upgrade Guide (direct quote)
"1-system upgrade, understandably, must always be done on the same platform as the earlier release of the Enterprise Manager system because you are upgrading on the same host.

2-system upgrade is possible on the same platform as the earlier release of the Enterprise Manager system, and from Linux (32-bit) to Linux (64-bit) (or vice versa) because you are upgrading on a remote host.

1-system upgrade on a different host is possible on the same platform as the earlier release of the Enterprise Manager system, and from Linux (32-bit) to Linux (64-bit) (or vice versa) because you are upgrading on a remote host, much like 2-system upgrade approach.

Comparison of the three approaches.

This blog uses 1-system upgrade on the same host path. The upgrade consists of four main steps. Installing the pre-upgrade patch which allows the agent upgrade. Upgrade of agents and then finally grid control upgrade and post-upgrade steps. Oracle documentation provides a detail breakdown of these steps.

There are few things to know about the upgrade.

Upgrades do not upgrade the database, if the existing database is certified for grid control see 412431.1 then it shouldn't be an issue. For this DB is

12c Agents communicate only with 12c OMS so agents are upgrade before OMS and time between agent upgrade and OMS upgrade there won't be any communication between agent and OMS and there won't be any monitoring.

Any agent could be upgrades as long as 12c management software is available.

Do not upgrade EM cloud control on a NFS mounted drive.

EM cloud control will install jdk 1.6_24 and Oracle WebLogic Server 11g Release 1 (10.3.5) if these are not present. For this existing web logic version is 10.3.2
and jdk 1.6_30

Earlier agents ports are carried forward with the upgrade.

Get the pre-upgrade patch and apply it to OMS.
Oracle documents mention the pre-upgrade patch as 10044087 but this is now replaced with patch 13503844. So if the the download link prompt this patch it is fine. Follow the readme.txt in this patch bundle to install it. One of the pre-req for this patch is "It is Mandatory to have PSU1 (10065631) or later applied on your environment."

Metalink note How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? [ID 1358092.1] shows the latest patches available for OMS and agents for both unix and windows. At the time of this blog post the latest patch was 13248190 for OMS and 9346243 for Agent. So those were applied. Database had PSU applied.

Following metalink notes may help with PSU patch applying issues.
Troubleshooting the Improved Patching Solution for Grid Control 11.1GC Management Server [ID 1326515.1]
Grid Control 11g: post_install script for PSU 4 Patch 12423703 fails with error for "MGMT_SET_JOB_Q_PROC" [ID 1362945.1]
Grid Control 11g: Post Install Script For PSU Patch 12423703 gives Error: "ORA-20110:Unexpected error executing MGMT_SET_JOB_Q_PROC" [ID 1359157.1]
Applying Post Install Scripts For Patch Getting Error: "ORA-00068: invalid value 0 for parameter job_queue_processes" [ID 1375352.1]

Once the PSU patches applied it's time to apply the pre-upgrade console patch 13503844.
13503844]$ /opt/app/oracle/Middleware/oms11g/OPatch/opatch apply
Oracle Interim Patch Installer version
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/app/oracle/Middleware/oms11g
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/oracle/Middleware/oms11g/oraInst.loc
OPatch version    :
OUI version       :
Log file location : /opt/app/oracle/Middleware/oms11g/cfgtoollogs/opatch/13503844_Feb_09_2012_10_56_45/apply2012-02-09_10-56-44AM_1.log

OPatch detects the Middleware Home as "/opt/app/oracle/Middleware"

OPatch will do the following:
[Oracle Home discovery]                      : Configure and Validate Oracle Home info.
[Prerequisite for apply]                     : Invoke prerequisites to see if patch can be applied.
[Patch conflict detection for apply patch]   : Check if any conflict with already installed patches in Oracle Home.

Applying interim patch '13503844' to OH '/opt/app/oracle/Middleware/oms11g'
Verifying environment and performing prerequisite checks...
To continue, OPatch will do the following:
[Back up for restore]           : Back up 68 files for Oracle Home restore in case of patching error.
[Back up for rollback]          : Back up 39 files for future patch rollback.
[Patch Oracle Home for apply]   : Copy 39 file(s) to Oracle Home;
                                  Archive 4 jar file(s) with 254 class file(s).
[Update inventory for apply]    : Modify the inventory to record  patch [13503844].

Do you want to proceed? [y|n]
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y
Backing up files...

Patching component oracle.sysman.oms.core,

Verifying the update...
Patch 13503844 successfully applied
Log file location: /opt/app/oracle/Middleware/oms11g/cfgtoollogs/opatch/13503844_Feb_09_2012_10_56_45/apply2012-02-09_10-56-44AM_1.log

OPatch succeeded.

13503844]$ $ORACLE_HOME/bin/rcuJDBCEngine sys/gridb@ JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/puc_dblink_pkgdef.sql
Completed SQL script execution normally.
1 scripts were processed

13503844]$ $ORACLE_HOME/bin/rcuJDBCEngine sys/gridb@ JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/puc_dblink_pkgbody.sql
Completed SQL script execution normally.
1 scripts were processed

13503844]$ $ORACLE_HOME/bin/rcuJDBCEngine sysman/gridb321@ JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/pre_upg_console.sql
l_target_guids.COUNT : 4

oracle.sysman.beacon SMPADDON Oracle Beacon  Plugin
oracle.sysman.emrep SMPADDON Oracle Management And Repository Plugin
oracle.sysman.oh SMPADDON Oracle OracleHome
oracle.sysman.csa SMPADDON Oracle Client System Analyzer
oracle.sysman.db SMPADDON Oracle Database Plugin
oracle.sysman.emas SMPADDON Oracle Fusion Middleware Plugin
oracle.sysman.empa ADDON Oracle Siebel Plugin
oracle.sysman.vt ADDON Oracle Virtualization Plugin
oracle.fmw.bhv ADDON Oracle Beehive Plugin

Completed SQL script execution normally.
24 scripts were processed
When the patch is applied it add a link in the deployment tab page under upgrade topic.
Below is a screenshot of this page before pre-upgrade patch is applied.
Below is a screenshot of the same page after the pre-upgrade console patch applied.
This link will allow navigation into upgrade console page which has option to chose the upgrade type, agent upgrade status and over view page describing the steps in the upgrade. Below is the screenshot of the upgrade console page.
The overview link on this page gives information about each upgrade approach. Select the upgrade approach and click on the overview link. Below is screenshot of 1-system on same host and 1-system on different host overview page.

Download agent core plugin and other needed plugins
and copy these files into a directory in the host where grid control is running.

Below is the list of downloaded plugins.
-rw-r--r-- 1 oracle oinstall    495016 Feb  9 11:58
-rw-r--r-- 1 oracle oinstall    499849 Feb  9 11:58
-rw-r--r-- 1 oracle oinstall    297231 Feb  9 11:58
-rw-r--r-- 1 oracle oinstall   3925371 Feb  9 11:58
-rw-r--r-- 1 oracle oinstall 230699506 Feb  9 11:59
-rw-r--r-- 1 oracle oinstall 223979520 Feb  9 11:59
-rw-r--r-- 1 oracle oinstall 621251038 Feb  9 11:59
-rw-r--r-- 1 oracle oinstall 106452968 Feb  9 12:08
-rw-r--r-- 1 oracle oinstall 440206388 Feb  9 12:17
Select 1-system upgrade type and then click mange software link in the upgrade console page. This will show the agent upgradeability. without the above plugin being discovered agents will be shown as unable to upgrade. Below screenshots show the current state of agent upgradeability as no plugin or agent software is validated.

Give the directory location of the software and validate.

When the plugins and agent software is validate all the agents become upgradable. Resolve any issues, missing plugins and re-validate before continuing.

As part of the validate process plugins will be extracted into plugin_bin folder.
plugin_bin]$ ls -lrt
drwxr----- 3 oracle oinstall  4096 Feb  9 12:18 oracle.sysman.oh
drwxr----- 3 oracle oinstall  4096 Feb  9 12:18 oracle.sysman.emrep
drwxr----- 3 oracle oinstall  4096 Feb  9 12:18 oracle.sysman.emas
drwxr----- 3 oracle oinstall  4096 Feb  9 12:18 oracle.em.soee
drwxr----- 3 oracle oinstall  4096 Feb  9 12:18 oracle.sysman.beacon
drwxr----- 3 oracle oinstall  4096 Feb  9 12:18 oracle.sysman.db
drwxr----- 3 oracle oinstall  4096 Feb  9 12:19 oracle.sysman.csa
Clicking on the upgradable link next to each green pie chart will show the upgradeability of each agent and each target.

Once all agents are upgradeable click on "Agent Upgrade Status" link which is under "Other Links" in upgrade console page.At this stage all the status should be not started.

Click deploy and configure agents button. Multiple agents could be selected to be deployed and configured but this require host and agent credentials if these are not set then oracle and root credentials must be same for all hosts. Here agents are deployed and configured one at a time. Use credentials, selecting sudo gave error "sudo is not present".

Select the host deploy and configure the new agent.

Clicking the deploy and configure button open up the page below which allows to specify the new directory for agent host credentials.

Click continue and give root user credentials.

When the submit button is clicked it will start the deploy and configure job.

As said earlier it is also possible to deploy and configure on multiple hosts. This is useful in RAC environments.

When the job has finished the status will be shown. If job fails inspect the logs which could be accessed through the links given as steps in the same page.

Once the agent deployment is successful run agent readiness check.

At the end of agent readiness check configuration and deployment status will be success.

At the end agent readiness check Health Check Status column has "report pending verification". This requires manually verifying it is alright to proceed. Click on view and verify health check report button and inspect the report and confirm if it is error free.

After this agent is ready for the switch.

This where the down time starts. When switch agent is started the current 11g agent will be stopped and disabled. The 12c new agent will be started but cannot report to 11g OMS until it's upgrade. Therefore until the OMS is upgraded there will be no monitoring. To keep this unmonitored time to a minimum carry out the steps before "switch agent" all the agents. Considering the agents listed in this blog agent on host hpc1 will not be switched until hpc4 and rac1 and rac2 all have completed their agent deployments and health checks.
Agent switch could be done one agent a time or multiple agents at the same time. Below are the screenshots from switching agents on all the RAC nodes at the same time.

Agent upgrade status after all the agents are switched.

When all the agents are switched to 12c agents and monitoring has effectively halted until OMS is upgraded. Status of the new agent states its up and ready but still unable to communicate with OMS.
./emctl status agent
Oracle Enterprise Manager 12c Cloud Control
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
Agent Version     :
OMS Version       : (unknown)
Protocol Version  :
Agent Home        : /opt/app/oracle/grid_agent/agent12g/agent_inst
Agent Binaries    : /opt/app/oracle/grid_agent/agent12g/core/
Agent Process ID  : 4072
Parent Process ID : 3935
Agent URL         : https://rac1.   .net:3872/emd/main
Repository URL    : https://hpc4.   .net:4900/empbs/upload
Started at        : 2012-02-09 16:26:48
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 82
Size of XML files pending upload(MB)         : 0.54
Available disk space on upload filesystem    : 13.59%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2012-02-09 16:32:58
Last successful heartbeat to OMS             : (none)

Agent is Running and Ready
To keep the un-monitored time to a minimum begin the OMS upgrade soon after the agent switch.

Part 2 shows the OMS upgrade.

Useful metalink notes

Upgrade Advisor: Enterprise Manager 1-System Upgrade from to [ID 328.1]
12c: Upgrading the Enterprise Manager on Linux x86_64 to with the 1-System Method [ID 1364931.1]
How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? [ID 1358092.1]