EMD upload error: uploadXMLFiles skipped

There are situations where agent upload command gives the following error

emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS
related errors.

To fix this issue

exec mgmt_admin.cleanup_agent('host:port'); 

host:port is the hostname and port shown in the agent status output

Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /opt/app/oracle/grid_agent/agent11g
Agent binaries    : /opt/app/oracle/grid_agent/agent11g
Agent Process ID  : 25049
Parent Process ID : 25027
Agent URL         : https://hpc1.domain.net:3872/emd/main/
Repository URL    : https://hpc4.domain.net:4900/em/upload

This command should be run on grid control database login as sysman. This will remove the remote host information from the grid control repository.

On the remote host itself run

rm -r /sysman/emd/state/*
rm -r /sysman/emd/collection/*
rm -r /sysman/emd/upload/*
rm /sysman/emd/lastupld.xml
rm /sysman/emd/agntstmp.txt
rm /sysman/emd/blackouts.xml
rm /sysman/emd/protocol.ini

Some files may not exists in the agent home.

Clear the agent state

$Agent_home/emctl clearstate agent

Secure the agent again

emctl secure agent

and start the agent

emctl start agent

Monitor new host being registered on the grid control target page. On the agent status output monitor the following lines for sucessful uploads

Last successful upload                     
Total Megabytes of XML files uploaded so far 
Number of XML files pending upload          
Size of XML files pending upload(MB)        
Last successful heartbeat to OMS

Removing Grid Control Agent from One Node Only in Silent Mode

This blog is for situation where grid control agent is removed from one node in the cluster.

1. Stop the agent on the remote host

2. Run the following on the node on which agent is being removed

/opt/app/oracle/grid_agent/agent11g/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/oracle/grid_agent/agent11g "CLUSTER_NODES={rac2}" -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/app/oracle/oraInventory

3. Deinstall agent on the node oracle doc says to use -forceDeinstall but installer complains about this.

oui/bin/runInstaller -silent  "REMOVE_HOMES={/opt/app/oracle/grid_agent/agent11g}" -deinstall -waitForCompletion -removeallfiles -local -forceDeinstall
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-12-15_12-28-38PM. Please wait ...Oracle Universal Installer, Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

The command line arguments '-forceDeinstall' are not valid options. Type 'runInstaller  -help' at the command line for instructions on appropriate command line usage.
SEVERE:The command line arguments '-forceDeinstall' are not valid options. Type 'runInstaller  -help' at the command line for instructions on appropriate command line usage.

Run without it and command execute without an error

oui/bin/runInstaller -silent  "REMOVE_HOMES={/opt/app/oracle/grid_agent/agent11g}" -deinstall -waitForCompletion -removeallfiles -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-12-15_12-29-00PM. Please wait ...Oracle Universal Installer, Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

Starting deinstall

Deinstall in progress (Wednesday, December 15, 2010 12:29:05 PM GMT)
Configuration assistant "Agent Deinstall Assistant" succeeded
Configuration assistant "Oracle Configuration Manager Deinstall" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Wednesday, December 15, 2010 12:30:20 PM GMT)
End of deinstallations
Please check '/opt/app/oracle/oraInventory/logs/silentInstall2010-12-15_12-29-00PM.log' for more details.

4. On each of the remaining nodes run installer with remaining node list.

./runInstaller -updateNodeList ORACLE_HOME=/opt/app/oracle/grid_agent/agent11g "CLUSTER_NODES={rac1}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/app/oracle/oraInventory
'UpdateNodeList' was successful.

5. Remove all targets on that node from the grid control target page.

Recover From a Clusterware Home Deletion

This blog is as a result of the outcome seen twice on node deletion step 14.
It is assumed that all the clusterware files inside the CRS_HOME has been deleted and at best only the directories are available without any files in them.

In this case a new clusterare installation is needed unless those clusterware homes (each home individually) are backed up using some OS utility (this option hasn't been tested).

Even though clusterware is newly installed oracle home and asm home is intact and are cluster aware. So what is required is to add the existing homes, database, instances and etc to the new clusterware and ocr to be managed by the cluster.

1. Remove startup commands from /etc/inittab

2. Remove symbolic links and init scripts from init.d and rc#.d

cd /etc/rc.d
for i in `find . -name S96*`; do rm -f $i; done
for i in `find . -name K19*`; do rm -f $i; done

3. Remove directory with ocr location and scr settings.

rm -rf /etc/oracle

4. Change permission on clusterware files (ocr, vote) location

chown oracle:oinstall /dev/raw/raw1
chown oracle:oinstall /dev/raw/raw2

5. Zero out the block or character devices used for ocr and vote disk storage

dd if=/dev/zero of=/dev/raw/raw1 bs=8192 count=1024
dd if=/dev/zero of=/dev/raw/raw2 bs=8192 count=1024

6. Backup the current inventory folder and create a new empty inventory folder (oraInventory) change the permission on it to 770 and oracle:oinstall

7. Remove the crs home and create a new directory with ownership oracle:oinstall. Execute runInstaller for the clusterware. If clusterware has been upgraded, started with 10.2 and was upgraded to 11.1 when files got deleted, then start from the initial version if ocr and vote disk location doesn't have enough space to accomodate the new version and upgrade afterwards to match the version before the files got deleted. Once the installation is complete attache the other oracle homes (ORACLE_HOME, ASM_HOME) to new inventory.

./runInstaller -silent -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME="oracle_home" "CLUSTER_NODES={rac1,rac2}"

Once the installation is completed add listener, asm, database and instances.

8. Add listener

srvctl add listener -n rac1 -o $ORACLE_HOME   
srvctl add listener -n rac2 -o $ORACLE_HOME
srvctl start listener -n rac1
srvctl start listener -n rac2

9. Add asm

srvctl add asm -n rac1 -i +ASM1 -o $ORACLE_HOME
srvctl add asm -n rac2 -i +ASM2 -o $ORACLE_HOME
srvctl start asm -n rac1
srvctl start asm -n rac2

10. Add database

srvctl add database -d racdb -o $ORACLE_HOME -p '+DATA/racdb/spfileracdb.ora'
srvctl modify database -d racdb -o $ORACLE_HOME -p '+DATA/racdb/spfileracdb.ora' -m domain.net

11. Add instance

srvctl add instance -d racdb -i racdb1 -n rac1
srvctl modify instance -d racdb -i racdb1  -s +ASM1

srvctl add instance -d racdb -i racdb2 -n rac2
srvctl modify instance -d racdb -i racdb2 -s +ASM2

12. Start the database

srvctl start database -d racdb