Wednesday, July 1, 2020

Can't locate Data/Dumper.pm in @INC When Using TFACTL

"Can't locate Data/Dumper.pm in @INC" error was encountered while executing tfactl and at times during installation. The environment setup and details are as follows.
OS
cat /etc/oracle-release
Oracle Linux Server release 7.7

uname -r
4.14.35-1902.10.2.1.el7uek.x86_64
Database setup is Oracle restart 19.7.

As the issue wasn't fully resolved even after a SR, it's important to consider the above configuration as issue wasn't happening in RedHat environments with single instance database setup.
The tfactl now comes bundle in Autonomous Health Framework (AHF, refer 2550798.1). Tfactl SR bundles are important specially when raising SR . At times SR will not move forward until a tfactl SR bundle is uploaded to the ticket. Therefore not being able to install or get a SR bundle using tfactl could add to resolution time (and to the frustration).
Tfactl could be installed both as root (in daemon mode) and as non-root user. While installing as non-root user the error was encountered during following step.
./ahf_setup -ahf_loc /opt/cx/tfa

AHF Installer for Platform Linux Architecture x86_64
AHF Installation Log : /tmp/ahf_install_86513_2020_06_24-08_59_24.log
Starting Autonomous Health Framework (AHF) Installation
AHF Version: 20.1.3 Build Date: 202004290950
AHF Location : /opt/cx/tfa/oracle.ahf
AHF Data Directory : /opt/cx/tfa/oracle.ahf/data
Extracting AHF to /opt/cx/tfa/oracle.ahf
Can't locate Data/Dumper.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/cx/tfa/oracle.ahf/tfa/bin /opt/cx/tfa/oracle.ahf/tfa/bin/common /opt/cx/tfa/oracle.ahf/tfa/bin/modules /opt/cx/tfa/oracle.ahf/tfa/bin/common/exceptions) at /opt/cx/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
BEGIN failed--compilation aborted at /opt/cx/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
AHF is deployed at /opt/cx/tfa/oracle.ahf
ORAchk is available at /opt/cx/tfa/oracle.ahf/bin/orachk
AHF binaries are available in /opt/cx/tfa/oracle.ahf/bin
AHF is successfully installed
Moving /tmp/ahf_install_86513_2020_06_24-08_59_24.log to /opt/cx/tfa/oracle.ahf/data/pre-prod/diag/ahf/
Even though error occurred the output says AHF was successfully installed. The installation issue was resolved by adding $OH/perl/bin to path.
export PATH=$ORACLE_HOME/perl/bin:$PATH
$ which perl
/opt/cx/app/oracle/product/19.x.0/dbhome_1/perl/bin/perl

$ ./ahf_setup -ahf_loc /opt/cx/tfa

AHF Installer for Platform Linux Architecture x86_64
AHF Installation Log : /tmp/ahf_install_87817_2020_06_24-09_02_05.log
Starting Autonomous Health Framework (AHF) Installation
AHF Version: 20.1.3 Build Date: 202004290950
AHF Location : /opt/cx/tfa/oracle.ahf
AHF Data Directory : /opt/cx/tfa/oracle.ahf/data
Extracting AHF to /opt/cx/tfa/oracle.ahf
Configuring TFA in Standalone Mode...
Build Version : 201300  Build Date : 202004290950

Discovering Nodes and Oracle Resources

.--------------------------------------------------------------------------------.
|                          Summary of TFA Configuration                          |
+----------------+---------------------------------------------------------------+
| Parameter      | Value                                                         |
+----------------+---------------------------------------------------------------+
| TFA Location   | /opt/cx/tfa/oracle.ahf/tfa                                    |
| Data Directory | /opt/cx/tfa/oracle.ahf/data/pre-prod/tfa      |
| Repository     | /opt/cx/tfa/oracle.ahf/data/repository                        |
| Diag Directory | /opt/cx/tfa/oracle.ahf/data/pre-prod/diag/tfa |
| Java Home      | /opt/cx/tfa/oracle.ahf/jre                                    |
'----------------+---------------------------------------------------------------'

.-----------------------------------------------------------------------------------------------------------------.
| Host                     | Status of TFA | PID | Port    | Version    | Build ID             | Inventory Status |
+--------------------------+---------------+-----+---------+------------+----------------------+------------------+
| pre-prod                 | RUNNING       | -   | OFFLINE | 20.1.3.0.0 | 20130020200429095054 | COMPLETED        |
'--------------------------+---------------+-----+---------+------------+----------------------+------------------'

AHF is deployed at /opt/cx/tfa/oracle.ahf
ORAchk is available at /opt/cx/tfa/oracle.ahf/bin/orachk
AHF binaries are available in /opt/cx/tfa/oracle.ahf/bin
AHF is successfully installed
Moving /tmp/ahf_install_87817_2020_06_24-09_02_05.log to /opt/cx/tfa/oracle.ahf/data/pre-prod/diag/ahf/

However, the running of tfactl failed with "Can't locate Data/Dumper.pm" no matter which perl binary was in the path. On OEL the default perl binary for non-root user (oracle) is out of following location.
which perl
/usr/bin/perl
Running tfactl with this default perl binary fails.
./tfactl diagcollect -srdc dbrac
Can't locate Data/Dumper.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/fc/tfa/oracle.ahf/tfa/bin /opt/fc/tfa/oracle.ahf/tfa/bin/common /opt/fc/tfa/oracle.ahf/tfa/bin/modules /opt/fc/tfa/oracle.ahf/tfa/bin/common/exceptions) at /opt/fc/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
BEGIN failed--compilation aborted at /opt/fc/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
On RHEL the default perl binary for non-root user is out of /bin. Setting this in path also fails to run tfactl.
export PATH=/bin:$PATH
$ which perl
/bin/perl

$ ./tfactl diagcollect -srdc dbrac
Can't locate Data/Dumper.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/cx/tfa/oracle.ahf/tfa/bin /opt/cx/tfa/oracle.ahf/tfa/bin/common /opt/cx/tfa/oracle.ahf/tfa/bin/modules /opt/cx/tfa/oracle.ahf/tfa/bin/common/exceptions) at /opt/cx/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
BEGIN failed--compilation aborted at /opt/cx/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
Running with perl out of ORALCE_HOME also fails.
which perl
/opt/cx/app/oracle/product/19.x.0/dbhome_1/perl/bin/perl

$ ./tfactl diagcollect -srdc dbrac
Can't locate Data/Dumper.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/cx/tfa/oracle.ahf/tfa/bin /opt/cx/tfa/oracle.ahf/tfa/bin/common /opt/cx/tfa/oracle.ahf/tfa/bin/modules /opt/cx/tfa/oracle.ahf/tfa/bin/common/exceptions) at /opt/cx/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
BEGIN failed--compilation aborted at /opt/cx/tfa/oracle.ahf/tfa/bin/tfactl.pl line 294.
Support note 2233429.1 matches the issue at hand but solution wasn't relavent in this case. For tfactl to work DataDumper.pm and Data/Dumper.pm must be available inside Oracle home. In this case both these files were available.
cd $ORACLE_HOME
find . -name *Dumper.pm
./perl/lib/5.28.1/x86_64-linux-thread-multi/Data/Dumper.pm
./perl/lib/site_perl/5.28.1/x86_64-linux-thread-multi/DBI/Gofer/Serializer/DataDumper.pm
At this point raised a SR. However, SR didn't go anwyhere as the only thing SR handler could do is ask if the above two files are available. On top of that gave wrong information as ahf must be run as root when infact 2550798.1 states "If you are unable to install as root, then you can install as the Oracle home owner". So had to give up on relying on SR support and find a resolution or a workaround.



Inside the AHF (oracle.ahf) the tfactl is availble in serveral locations.
 find . -name tfactl
./data/pre-prod/diag/tfa/tfactl
./data/pre-prod/diag/tfactl
./ahf/bin/tfactl
./tfa/bin/tfactl
./bin/tfactl
All of those tfactl executions failed with the same error as above. However, inside "TFA Location" there's tfactl.pl script. Running this perl script directly with the perl out of ORACLE_HOME worked without error. The default perl binary out of OS (/usr/bin/perl) failed.
which perl
/opt/cx/app/oracle/product/19.x.0/dbhome_1/perl/bin/perl

cd /opt/cx/tfa/oracle.ahf/tfa/bin

perl tfactl.pl diagcollect -srdc dbrac

TFA will collect diagnostics for the last 1 hour(s).
Please enter the time of the incident [YYYY-MM-DD HH24:MI:SS], or  to collect for the last 1 hour(s):

Scripts to be run by this srdc:
Components included in this srdc: OS CRS DATABASE CHMOS ASM ASMPROXY ASMIO ACFS RHP DBWLM

Collection Id : 20200624092805pre-prod

Detailed Logging at : /opt/cx/tfa/oracle.ahf/data/repository/srdc_dbrac_collection_Wed_Jun_24_09_28_08_GMT_2020_node_all/diagcollect_20200624092805_pre-prod.log
2020/06/24 09:28:13 GMT : NOTE : Any file or directory name containing the string .com will be renamed to replace .com with dotcom
2020/06/24 09:28:13 GMT : Collection Name : tfa_srdc_dbrac_Wed_Jun_24_09_28_08_GMT_2020.zip
2020/06/24 09:28:13 GMT : Collecting additional diagnostic information...
2020/06/24 09:28:13 GMT : Getting list of files satisfying time range [06/24/2020 08:28:13 GMT, 06/24/2020 09:28:13 GMT]
2020/06/24 09:28:13 GMT : Completed collection of additional diagnostic information...
2020/06/24 09:28:28 GMT : Collecting ADR incident files...
2020/06/24 09:28:28 GMT : Completed Local Collection
.----------------------------------------------------.
|                 Collection Summary                 |
+--------------------------+-----------+------+------+
| Host                     | Status    | Size | Time |
+--------------------------+-----------+------+------+
| pre-prod                 | Completed | 9MB  |  15s |
'--------------------------+-----------+------+------'

Logs are being collected to: /opt/cx/tfa/oracle.ahf/data/repository/srdc_dbrac_collection_Wed_Jun_24_09_28_08_GMT_2020_node_all
/opt/cx/tfa/oracle.ahf/data/repository/srdc_dbrac_collection_Wed_Jun_24_09_28_08_GMT_2020_node_all/pre-prod.tfa_srdc_dbrac_Wed_Jun_24_09_28_08_GMT_2020.zip
In summary even if the files mentioned in 2233429.1 are present, the tfactl script would fail to run. In such cases run the tfactl perl script directly.

Useful MOS notes
Installing or Running TFA fails with "Can't locate Data/Dumper.pm in @INC (@INC contains" error [ID 2233429.1]