Tuesday, March 17, 2020

Installing Enterprise Manager Cloud Control 13c (13.4)

This post shows the summary of steps for installing cloud control 13.4. There's an earlier post showing detail installation of cloud control 13.3. Installation of 13.4 is very similar if not identical to that of the installation of 13.3. Hence this post only shows the difference between the two installation. If installing 13.4 cloud control then refer this post in-conjunction with earlier post on 13.3.
One of the main changes is that 13.4 now support 12.2, 18c and 19c as the repository storage DB. For this installation a 19c (19.6) database was created with only the following components installed.
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0

SQL> select comp_id,status from dba_registry;

COMP_ID                        STATUS
------------------------------ --------------
CATALOG                        VALID
CATPROC                        VALID
RAC                            OPTION OFF
XDB                            VALID
OWM                            VALID
CONTEXT                        VALID
Once the database that will be used as the repository is created then further parameter changes are needed before EM installation could proceed. In 13.3 installation if the database did not meet the pre-reqs then a details of those pre-reqs that aren't met are shown on the OUI. This is not the case with 13.4 installation. In 13.4 a general message will be shown as below.

Exact pre-reqs that failed could be found on the log files available at the temporary directory specified at the beginning of the EM installation (more on this location later). Assuming /opt/tmp was given as the temporary directory then log files will be available in /opt/tmp/OraInstall2020-02-27_03-06-40PM/emdbprereqs/LATEST/componentLog and the log files to look for are repos_perf.log and repository.log.
Following if the list of parameters that were changed so the installation could proceed.
alter system set "_allow_insert_with_update_check"=true scope=both;
alter system set "_optimizer_nlj_hj_adaptive_join"= FALSE scope=both sid='*';
alter system set "_optimizer_strans_adaptive_pruning" = FALSE scope=both sid='*';
alter system set "_px_adaptive_dist_method" = OFF scope=both sid='*';
alter system set "_sql_plan_directive_mgmt_control" = 0 scope=both sid='*';
alter system set "_optimizer_dsdir_usage_control" = 0 scope=both sid='*';
alter system set "_optimizer_use_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_gather_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_performance_feedback" = OFF scope=both sid='*';

alter system set parallel_min_servers=0 scope=both;
alter system set parallel_max_servers=8  scope=both;
alter system set sga_max_size=7G scope=spfile;
alter system set sga_target=7G scope=spfile;
alter system set session_cached_cursors=500 scope=spfile;
alter system set shared_pool_size=600M scope=spfile;
Also manually disable the automatic stat gathering before the installation. This will be enabled by EM installation in the end. So no need to manually enable it.
exec DBMS_AUTO_TASK_ADMIN.DISABLE(client_name => 'auto optimizer stats collection',operation => NULL,window_name => NULL);
Finally similar to 13.3, the redo file size must be at least 600M.
    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME     CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
         1          1          9  629145600        512          1 NO  INACTIVE                624704 27-FEB-20       624707 27-FEB-20          0
         2          1         11  629145600        512          1 NO  CURRENT                 726543 27-FEB-20   9.2954E+18                    0
         3          1         10  629145600        512          1 NO  INACTIVE                624707 27-FEB-20       726543 27-FEB-20          0
The start of the installation will fail if the default temporary location doesn't have enough storage. The installer will prompt for another location. But even a location with sufficient storage is provided the installation will fail to run.
./em13400_linux64.bin 
ERROR: Temporary directory /tmp does not have enough free space. At least 12289 MB of free space are required.
Please input another directory or [Exit]: /opt/tmp
ERROR: Cannot find component oracle.jdk.
ERROR: Cannot find components specified in JRE_COMPONENT=oracle.jdk,oracle.jre.
ERROR: Installer execution failed (1).
To fix this provide an alternative temp directory location along side the installer as below (refer 2637530.1).
./em13400_linux64.bin  -J-Djava.io.tmpdir=/opt/tmp
As said earlier the installation steps are same as 13.3 as such they are not shown here. Summary of the installation is shown below.

When prompted run the root script.
# /opt/app/software/em/middleware/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /opt/app/software/em/middleware/root.sh ......
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/software/em/middleware

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
/etc exist
/opt/app/software/em/middleware

Finished product-specific root actions.
/etc exist
Finished execution of  /opt/app/software/em/middleware/root.sh ......

Starting to execute /opt/app/software/em/agent/agent_13.4.0.0.0/root.sh ......

Finished product-specific root actions.
/etc exist
Finished execution of  /opt/app/software/em/agent/agent_13.4.0.0.0/root.sh ......
Installation summary.




Adding Targets on 13.4

There's an earlier post on adding targets on 13.3. Steps for adding targets on 13.4 is also the same. However, there were few issues unseen on 13.3 when adding targets. This section list those issues and the solutions for them.
Even though the sudo command list mentioned on 13.3 add target was used for 13.4, the agent deploying gave the following warning.

This issue could be resolved in two ways. One is to by unchecking the "!" in the visiblepw line in /etc/sudoers file. From
Defaults   !visiblepw
to
Defaults   visiblepw
Second method is to use -S option in the previlege delegation setting (refer 1490407.1). This option was used for agent deployment in 13.4. Use of -S option is shown below.


If the listener control is restricted to IPC protocol then agent may not be able to determine the listener status without some additional work. On the EM console listener will be listed as down and agent log will have following line.
TNS-01194: The listener command did not arrive in a secure transport
To resolve this modify the listener monitor configuration on the EM console by speechifying the IPC key.

To avoid bug 22088515 grant execute on dbms_lock to dbsnmp on the target database and reload the agent (Refer 2366263.1).
SQL> GRANT EXECUTE ON sys.dbms_lock TO dbsnmp;

Grant succeeded.

 ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.4.0.0.0
OMS Version            : 13.4.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/agent/agent_inst
Agent Log Directory    : /opt/agent/agent_inst/sysman/log
Agent Binaries         : /opt/agent/agent_13.4.0.0.0
Core JAR Location      : /opt/agent/agent_13.4.0.0.0/jlib
Agent Process ID       : 10845
Parent Process ID      : 10808
Agent URL              : https://rhel71.domain.net:3872/emd/main/
Local Agent URL in NAT : https://rhel71.domain.net:3872/emd/main/
Repository URL         : https://hpc8.domain.net:4903/empbs/upload
Started at             : 2020-03-02 10:26:24
Started by user        : asanga
Operating System       : Linux version 3.10.0-957.el7.x86_64 (amd64)
Number of Targets      : 10
Last Reload            : (none)
Last successful upload                       : 2020-03-05 12:58:05
Last attempted upload                        : 2020-03-05 12:58:05
Total Megabytes of XML files uploaded so far : 5.09
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 8.61%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2020-03-05 12:57:48
Last successful heartbeat to OMS             : 2020-03-05 12:57:48
Next scheduled heartbeat to OMS              : 2020-03-05 12:58:48

---------------------------------------------------------------
Agent is Running and Ready
[asanga@rhel71 bin]$ ./emctl reload agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully

[asanga@rhel71 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.4.0.0.0
OMS Version            : 13.4.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/agent/agent_inst
Agent Log Directory    : /opt/agent/agent_inst/sysman/log
Agent Binaries         : /opt/agent/agent_13.4.0.0.0
Core JAR Location      : /opt/agent/agent_13.4.0.0.0/jlib
Agent Process ID       : 10845
Parent Process ID      : 10808
Agent URL              : https://rhel71.domain.net:3872/emd/main/
Local Agent URL in NAT : https://rhel71.domain.net:3872/emd/main/
Repository URL         : https://hpc8.domain.net:4903/empbs/upload
Started at             : 2020-03-02 10:26:24
Started by user        : asanga
Operating System       : Linux version 3.10.0-957.el7.x86_64 (amd64)
Number of Targets      : 10
Last Reload            : 2020-03-05 12:58:26
Last successful upload                       : 2020-03-05 12:58:05
Last attempted upload                        : 2020-03-05 12:58:05
Total Megabytes of XML files uploaded so far : 5.09
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 8.61%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2020-03-05 12:57:48
Last successful heartbeat to OMS             : 2020-03-05 12:57:48
Next scheduled heartbeat to OMS              : 2020-03-05 12:58:48

---------------------------------------------------------------
Agent is Running and Ready

Useful Metalink notes
EM 13.4 : Input of Different Temp Location fails with 'Cannot find component oracle.jdk.' [ID 2637530.1]
EM 12c, EM 13c: Why does Agent execute "BEGIN Dbms_lock.sleep(60)" statement Having SQL ID "g0bggfqrddc4w" in the Target Database ? [ID 2366263.1]
EM12c EM13c: Metric Collection Error for Listener Target TNS-12564: TNS:connection refused TNS-01194: The listener command did not arrive in a secure transport [ID 2296762.1]

Related Posts
Upgrading Enterprise Manager Cloud Control from 13.4 to 13.5 - 1
Adding Targets on EM Cloud Control 13c
Installing Enterprise Manager Cloud Control 13c (13.3)
Installing Grid Control 11gR1 and Deploying Agents
Upgrading Grid Control 11g to 12c - 1