Wednesday, December 15, 2010

Installing Grid Control 11gR1 and Deploying Agents

Installing Enterprise Manager Grid Control has three main parts.
1. Installing Weblogic for the webtier of the grid control
2. Installing the database
3. Finally the installing of actual grid control software

Some of the information is directly copied from the Enterprise Manager Grid Control Basic Installation Guide and shown in italic

1. Install Java before installing the weblogic. For Linux Platform (32/64 Bit)JDK version must be SUN JDK 1.6_18 or higher. JRockit is not supported.

2. Install Weblogic. For 64bit version download the generic version of the installation, which is a jar file (eg. wls1032_generic.jar)

There are serveral requirments before installing weblogic.

Ensure that Oracle WebLogic Server 10.3.2 (Oracle Fusion Middleware 11g Release 1 Patch Set 1) is already installed on the host where you plan to install Enterprise Manager Grid Control.

Ensure that the installation was a typical installation, and even if it was a custom installation, ensure that components chosen for custom installation were the same as the ones associated with a typical installation.

Ensure that the installation was under the Middleware Home directory. For example, /opt/app/oracle/Middleware/wlserver_10.3

Ensure that no other Oracle Fusion Middleware products or components are installed in the Middleware Home directory where Oracle WebLogic Server 10.3.2 is installed.


3. Patch weblogic with patch ID WDJ7. This patch fixes bugs 8990616, 9100465, and 9221722. For information on applying this patch, see My Oracle Support note 1072763.1 How to Download and Apply the Recommended WLS patch WDJ7 on WLS for 11g Grid Control Installation or Upgrade
Oracle Smart Update utility must be used for this, which could be run with Middleware_Home/utils/bsu/bsu.sh, it requires metalink login download the patch.

Select the patch and downloadOnce the patch is downloaded it will be available on the downloaded sectionClick Manage Patches tab and apply the patchFurther information on patches and activation and weblogic documentation

4. Install the database software and create the database. The database software could be 11.1 or 11.2 but requires some patching.

Ensure that the existing, certified Oracle Database is one of the databases listed in My Oracle Support note 412431.1. The database can reside either on the host where you are installing the product or on a remote host.

(Optional) If you are installing using Oracle Database 11g Release 1 (11.1.0.7.0), then ensure that you apply the patch for bug# 9066130.
This patch only available for 11.1.0.7.1 anything higher would have the patch on the psu so no need to apply separately.

(Optional) If you are installing using Oracle Database 11g Release 2 (11.2.0.1.0), then ensure that you apply the patch for bug# 9002336 and 9067282.

Ensure that existing, certified Oracle Database is not in QUIESCE mode.

Ensure that your existing, certified Oracle Database does NOT have Database Control SYSMAN schema. If it has, that is, if your existing database is configured with Database Control, then deconfigure it.

Best option create the database using dbca without selecting enterprise manager option. Still sysman schema will be there but the account is expired and lock. Drop the sysman schema and em with
emca -deconfig dbcontrol db -repos drop -SYS_PWD gridb -SYSMAN_PWD gridb
If a sysman scheam is there installation of grid control will fail.

Ensure that the fine-grained access control option is set to TRUE in the existing, certified Oracle Database so that the Management Repository can be created. To verify this, run the following command:
select value from v$option where parameter = 'Fine-grained access control';
Make the temporary tablespace and undo tablespace auto extensible.

If the database is not in archive log mode change to archive log mode

5. Execute the runInstaller for grid control software. Create the oms directory under the middleware directory (eg. /opt/app/oracle/Middleware/oms11g) Post installation phaseFinish page will show the oms url and the admin url
6. After the installation ends successfully, the OMS and the Management Agent start automatically these could be checked with
oms status check
/opt/app/oracle/Middleware/oms11g/bin/emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
agent status check
/opt/app/oracle/Middleware/agent11g/bin/emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /opt/app/oracle/Middleware/agent11g
Agent binaries    : /opt/app/oracle/Middleware/agent11g
Agent Process ID  : 21415
Parent Process ID : 21389
Agent URL         : https://hpc4.domain.net:3872/emd/main/
Repository URL    : https://hpc4.domain:4900/em/upload
Started at        : 2010-12-07 17:37:03
Started by user   : oracle
Last Reload       : 2010-12-07 17:37:25
Last successful upload                       : 2010-12-08 10:23:24
Total Megabytes of XML files uploaded so far :    88.26
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    52.55%
Last successful heartbeat to OMS             : 2010-12-08 10:26:51
---------------------------------------------------------------
Agent is Running and Ready
agent upload check
/opt/app/oracle/Middleware/agent11g/bin/emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
7. Login to the grid control as sysman and configure the Oracle Database and Oracle Automatic Storage Management (Oracle ASM) target for monitoring.

In the Enterprise Manager Grid Control console, when you view the Home page for an Oracle Database target for the first time, the Database Home page may not display any monitoring data, and the status of the database may indicate that there is a metric collection error. This is because the DBSNMP credentials has not been configured, or has been locked due to unsuccessful login attempts.

Similarly, when you view the Home page of an Oracle Automatic Storage Management (Oracle ASM) target for the first time, the status of the Oracle ASM instance may be unknown or unavailable, and the Home page may indicate that the Management Agent is unavailable (down). Again, this is because you have not specified the ASM SYS credentials.

To fix this problem for an Oracle Database target, If your DBSNMP user account is locked, then to unlock it follow these steps:

In Grid Control, click Targets and then Databases.
On the Databases page, from the table that lists all databases, click a database name.
On the Database Home page, click the Server tab.
On the Server page, from the Security section, click Users. If you are prompted to log in to the database, make sure to use a database user account with DBA privileges such as SYSTEM.
On the Users page, find and select the DBSNMP user account. From the Actions list, select Unlock User, and click Go. If you are asked to confirm whether you want to unlock the DBSNMP user account, click Yes.


8. For accessing the Enterprise Manager Grid Control console, ensure that you use only certified browsers as mentioned in My Oracle Support note 412431.1

9. Adding agent to monitor remote targets. Several pre-reqs must be satistifed before agents could be deployed to remote hosts.
If you want to view the status of an installation session that was previously run, then click Agent Installation Status on the Deployments page. However, do not attempt to view the installation status until the installation is complete. If you do, you will see an error.

Ensure that you install the Management Agent only on certified operating systems as mentioned in My Oracle Support note 412431.1.

Ensure that you allocate 100 MB of space for the central inventory directory.

Also ensure that the central inventory directory is not on a shared file system. If it is already on a shared file system, then switch over to a non-shared file system by following the instructions outlined in My Oracle Support note 1092645.1

Ensure that the installation base directory you specify is empty and has write permission.

If you want to install Oracle Management Agent 10g, then ensure that you download that Management Agent software.

If you want to install Oracle Management Agent 11g Release 1 for an operating system that is different from the one on which the Oracle Management Service 11g Release 1 is running, then ensure that you download the Management Agent software for that operating system.



Validate the path to all command locations in oms11g/sysman/agent_download/11.1.0.1.0/linux_x64/agentdeploy (Paths.properties,sPaths.properties,ssPaths_.properties,userPaths.properties).
On linux 64 sudo path is /usr/bin/sudo but it is listed on ssPaths_*** file as /usr/local/bin/sudo. change this.

Ensure that the host names and the IP addresses are properly configured in the /etc/hosts file. For example, for installing a Management Agent on a host, ensure that the host name specified in the /etc/hosts file is unique, and ensure that it maps to the correct IP address of that host. Otherwise, the installation can fail on the product-specific prerequisite check page.The installation uses hostnames instead of IPs even if IPs are specified during the setup. Therefore it's better to have hostnames of the remote hosts in /etc/hosts on the gird control server.

If the destination host and the host on which OMS is running belong to different network domains, then ensure that you update the /etc/hosts file on the destination host to add a line with the IP address of that host, the fully-qualified name of that host, and the short name of the host.

If the central inventory owner and the user installing the Management Agent are different, then ensure that they are part of the same group.
Also ensure that the inventory owner and the group to which the owner belongs have read and write permissions on the inventory directory.
For example, if the inventory owner is abc and user installing the Management Agent is xyz, then ensure that abc and xyz belong to the same group, and they have read and write access to the inventory.

Ensure that you have read, write, and execute permissions on oraInventory on all remote hosts. If you do not have these permissions on the default inventory (typically at /etc/oraInst.loc) on any remote host, then you can specify the path to an alternative inventory location by using one of the following options in the Additional Parameters section of the Agent Deployment Wizard:
On the Installation Details page, if you select Default, from Management Server location, which means the software is on the OMS host, then use the -i option.
On the Installation Details page, if you select Another Location, which means the software is in any other location other than the OMS host, then use the -invPtrLoc option.

Ensure that the SSH daemon is running on the default port (that is, 22) on all the destination hosts. If the port is a non-default port, that is, any port other than 22, then update the SSH_PORT property in the following file that is present in the OMS Instance Base:
/sysman/prov/resources/Paths.properties

Ensure that the PubkeyAuthentication parameter is enabled in the sshd_config file.
To verify the value of this parameter, run the following command:
grep PubkeyAuthentication /sshd_config.
Default is yes and above command will show yes but the line is commented, if it's not explicitly set to no then no additional work is required.

Add sudo privilege on the remote host to oracle this could be removed once the agent is installed. This would make the installation run smoother and allow the running of the root.sh when agents are deployed to remote hosts. add the following line to /etc/sudoers
oracle  ALL=(ALL)       NOPASSWD:ALL
and comment out the line
Defaults    requiretty
For any problems refer metalink note 363509.1 Problem - Agent Installation Using Push Method Fails with Error 'User not enabled for sudo' when sudo has been configured correctly

From the host where grid control is installed it should be possible to ping the host(s) (cluster nodes etc) and get a response. If ping response is blocked (ie firewall) then agent deployment will fail with a message remote host is down.

10. To install agent from the grid control hom page select deployment tab -> install agent -> fresh install. New agent deployment page will open.
Select the agent version and platform, enter the user details, root.sh could be run after the instllation before the post install scripts if sudo is not setup. To run root.sh the user entered must have sudo privilege.

For a agent deployment onto a RAC specify all the nodes that are part of the cluster or subset of nodes if agents are not deployed on some nodes.

Once installation is completed agent status on the remote host would be as
emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /opt/app/oracle/grid_agent/agent11g
Agent binaries    : /opt/app/oracle/grid_agent/agent11g
Agent Process ID  : 22954
Parent Process ID : 22917
Agent URL         : http://hpc1.domain.net:3872/emd/main/
Repository URL    : HTTPS://hpc4.domain.net:4900/em/upload/
Started at        : 2010-12-08 12:37:27
Started by user   : oracle
Last Reload       : 2010-12-08 12:37:27
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :       23
Size of XML files pending upload(MB)         :    11.19
Available disk space on upload filesystem    :     9.98%
Last attempted heartbeat to OMS              : 2010-12-08 12:40:59
Last successful heartbeat to OMS             : unknown
---------------------------------------------------------------
Agent is Running and Ready
Secure the agent with
emctl secure agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password :
Agent successfully restarted...   Done.
Securing agent...   Successful.
The password is the agent registration password used during the installation of grid software. after the securing is done agent communication will begin.
emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /opt/app/oracle/grid_agent/agent11g
Agent binaries    : /opt/app/oracle/grid_agent/agent11g
Agent Process ID  : 25049
Parent Process ID : 25027
Agent URL         : https://hpc1.domain.net:3872/emd/main/
Repository URL    : https://hpc4.domain.net:4900/em/upload
Started at        : 2010-12-08 12:42:43
Started by user   : oracle
Last Reload       : 2010-12-08 12:42:43
Last successful upload                       : 2010-12-08 12:42:49
Total Megabytes of XML files uploaded so far :     6.02
Number of XML files pending upload           :       40
Size of XML files pending upload(MB)         :    29.29
Available disk space on upload filesystem    :     9.95%
Last successful heartbeat to OMS             : 2010-12-08 12:42:45
---------------------------------------------------------------
Agent is Running and Ready
When agent communcation begin the target database will auto appear on targe page (page refresh might be required)

11. Add the target database by enabling dbsnmp
Set the monitoring credentials for a standalone Oracle Database or Oracle RAC database. To do so, follow these steps:
In Grid Control, click Targets and then Databases.
On the Databases page, find and select the database target and click Monitoring Configuration.
On the Properties page, specify the password for the DBSNMP user in the Monitor Password field. To verify the monitoring credentials, click Test Connection.
If the connection is successful, click Next, then click Submit.


For ASM
Set the monitoring credentials for Oracle ASM. To do so, follow these steps:
In Grid Control, click Targets and then Databases.
On the Databases page, find and select the Oracle ASM target and click Monitoring Configuration.
On the Properties page, specify the password for the ASMSYS user in the Password field. To verify the monitoring credentials, click Test Connection.
If the connection is successful, click Next, then click Submit.


In cluster database the domain name is added to short vip names, which may not be in the /etc/hosts, in this case cluster db status may not get updated. Edit the listener host to reflect the real vip host name.

12. Stop and Start of Grid Control. The stop sequence stop agent, oms and db. The start sequence is start db, oms and agent.

stop agent
/opt/app/oracle/Middleware/agent11g/bin/emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
stop oms
/opt/app/oracle/Middleware/oms11g/bin/emctl stop oms -all
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Then stop the the database.

Starting oms, start database if not started
/opt/app/oracle/Middleware/oms11g/bin/emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up
start agent
/opt/app/oracle/Middleware/agent11g/bin/emctl start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting agent ...... started.
Some useful metalink notes

How to Install Web Logic Server 10.3.2 for Installing 11g Grid Control [ID 1063762.1]
Step by Step Installation of Enterprise Manager Grid Control 11.1.0.1 [ID 1059516.1]
Grid Control 11g: Example for Installing WebLogic Server 10.3.2 on OEL 5.3 x86_64 [ID 1063112.1]
Case Study - Installing Grid Control 11.1.0.1 - Installation of jdk1.6 0n Linux x86_64 Before Installing WebLogic Server 10.3.2 [ID 1063587.1]
Grid Control 11g: How to Install 11.1.0.1.0 on OEL5.3 x86_64 with a 11.1.0.7.0 Repository Database [ID 1064495.1]
Required External Components and Versions for the Grid Control 11.1.0.1.0 Installation [ID 1106105.1]
Master Note for Grid Control 11.1.0.1.0 Installation and Upgrade [ID 1067438.1]
Required Patches for Grid Control 11g (11.1.0.1.0) [ID 1101208.1]
How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes? [ID 1358092.1]
Oracle Enterprise Manager Grid Control Certification Checker [ID 412431.1]