Showing posts with label hostname. Show all posts
Showing posts with label hostname. Show all posts

Monday, April 15, 2019

DIA-49802: missing read, write, or execute permission on specified ADR home directory

Executing an 18c (18.5) Oracle restart reconfiguration after a hostname change resulted in following error.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-25-04AM.log
2019/03/27 08:25:06 CLSRSC-363: User ignored prerequisites during installation
Oracle Clusterware infrastructure error in OCRCONFIG (OS PID 8212): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

Oracle Clusterware infrastructure error in CLSCFG (OS PID 8223): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:26:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

2019/03/27 08:28:17 CLSRSC-214: Failed to start the resource 'ohasd'
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2019-03-27 08:26:28.832
[client(8848)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8848
2019-03-27 08:26:28.845
[client(8848)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:31.155
[client(8900)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8900
2019-03-27 08:26:31.168
[client(8900)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:33.140
[client(8950)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8950
2019-03-27 08:26:33.152
[client(8950)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:35.195
[client(9000)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9000
2019-03-27 08:26:35.208
[client(9000)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:37.231
[client(9050)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9050
2019-03-27 08:26:37.243
[client(9050)]CRS-2112:The OLR service started on node ip-172-31-1-113.

2019/03/27 08:28:17 CLSRSC-318: Failed to start Oracle OHASD service
Died at /opt/app/oracle/product/18.0.0/grid/crs/install/crsinstall.pm line 3226.

It appears during the reconfiguration, which is executed as root user the folders inside $ORACLE_BASE/diag/crs/`hostname -s`/crs gets created with root ownership but are written to as oracle (or grid if role separation is used) user.
cd /opt/app/oracle/diag/crs/ip-172-31-1-113/crs
[root@ip-172-31-1-113 crs]# ls -l

drwxrwx---. 2 root root  4096 Mar 27 08:24 alert
drwxrwx---. 2 root root  4096 Mar 27 08:24 cdump
drwxrwx---. 2 root root  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 root root  4096 Mar 27 08:24 lck
drwxrwx---. 4 root root  4096 Mar 27 08:29 log
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 root root  4096 Mar 27 08:24 stage
drwxrwx---. 2 root root  4096 Mar 27 08:24 sweep
drwxrwx---. 2 root root 12288 Mar 27 08:33 trace


This issue only happens during reconfiguration. On a new installation the ownership is set to the user installing the grid software, either oracle or grid. Changing the ownership to oracle (or grid) user in these directories resolved the issue.
chown -R oracle:oinstall *

[root@ip-172-31-1-113 crs]# ls -l
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 alert
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 cdump
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incident
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 lck
drwxrwx---. 4 oracle oinstall  4096 Mar 27 08:29 log
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 stage
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 sweep
drwxrwx---. 2 oracle oinstall 20480 Mar 27 08:41 trace
Re-run the Oracle restart configuration again.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-38-56AM.log
2019/03/27 08:38:59 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:39:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113'
CRS-2673: Attempting to stop 'ora.evmd' on 'ip-172-31-1-113'
CRS-2677: Stop of 'ora.evmd' on 'ip-172-31-1-113' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

ip-172-31-1-113     2019/03/27 08:42:19     /opt/app/oracle/product/18.0.0/grid/cdata/ip-172-31-1-113/backup_20190327_084219.olr     2532936542
2019/03/27 08:42:53 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Related Post
Changing Hostname in a Standalone DB Configuration with ASM

Tuesday, March 1, 2016

Changing Hostname in a Standalone DB Configuration with ASM

Grid infrastructure software is required to setup ASM in a standalone DB configuration. GI has hostname dependent directory paths and components. Changing the hostname in a such configuration requires re-configuring the GI software after the hostname has been changed. This post list the steps for changing the hostname in a standalone DB configuration with ASM.
The current hostname is rhel7.domain.net and this will changed to rhel7s.domain.net
In certain systems (in this case OS RHEL 7.0, kernel 3.10.0-123.el7.x86_64, Oracle 11.2.0.4) may run in to following error due to perl version related issues. Solution for this is to use the perl binary provided with GI.
$GI_HOME/crs/install/roothas.pl -deconfig -force
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
The MOS notes mention to change the hostname and then de-configure but this fails with the following error
export ORACLE_HOME=$GI_HOME
$GI_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Can't open /etc/oracle/scls_scr/rhel7s/grid/ohasdstr for write: No such file or directory at /opt/app/oracle/product/11.2.0/grid/crs/install/s_crsconfig_lib.pm line 1386.
Reason is that deconfig option is looking for a directory path that does not exists
# ls /etc/oracle/scls_scr/
rhel7
It maybe possible to rename this directory and run the deconfig. But instead the deconfig was run (as root) without changing the hostname (reverting to original hostname).
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
CRS resources for listeners are still configured
PRKO-2573 : ONS daemon is already stopped.
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel7'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel7'
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel7'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel7' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel7' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel7' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel7'
CRS-2677: Stop of 'ora.asm' on 'rhel7' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel7'
CRS-2677: Stop of 'ora.cssd' on 'rhel7' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7'
CRS-2677: Stop of 'ora.evmd' on 'rhel7' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack
Once the deconfig is complete change the hostname
hostname
rhel7s.domain.net
Run (as root) the roothas.pl script to configure the HAS again.
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel7s successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

rhel7s     2016/02/12 13:10:13     /opt/app/oracle/product/11.2.0/grid/cdata/rhel7s/backup_20160212_131013.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
At this stage the services would be in the following states
Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.cssd                            ora.cssd.type                  OFFLINE            OFFLINE
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
As grid user, enable the cssd auto start and stop and start the HAS again
$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
$ crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7s'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7s'
CRS-2677: Stop of 'ora.evmd' on 'rhel7s' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7s' has completed
CRS-4133: Oracle High Availability Services has been stopped.
$ crsctl start has
CRS-4123: Oracle High Availability Services has been started.
Check the states of the services and HAS service
Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE

$ crsctl check has
CRS-4638: Oracle High Availability Services is online
GI has its services but listener, ASM and DB are missing. These need to be manually added to the configuration. First add and start the listener
$ srvctl add listener
$ srvctl start listener

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE


Add the ASM and set auto start option to 1. In this case the ASM Spfile was not residing in a ASM diskgroup.
$ srvctl add asm
$ crsctl modify resource "ora.asm" -attr "AUTO_START=1"
$ srvctl modify asm -p /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora -l listener

$ srvctl config asm
ASM home: /opt/app/oracle/product/11.2.0/grid
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

$ crsctl stop has
$ crsctl start has

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
Finally add the database and start it
$ srvctl add database -d std11g2 -o $ORACLE_HOME -p +DATA/std11g2/spfilestd11g2.ora
$ srvctl config database -d std11g2
Database unique name: std11g2
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/std11g2/spfilestd11g2.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: std11g2
Disk Groups:
Services:
$ srvctl start database -d std11g2

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type              ONLINE             ONLINE             rhel7s
This conclude the changing of hostname in a standalone DB configuration with ASM.

Useful metalink notes
How to Reconfigure Oracle Restart on 12c / 12.1 [ID 1570358.1]
rootcrs.pl/roothas.pl Fails With Can't locate Env.pm [ID 2019784.1]
rootcrs.pl/roothas.pl Fails With "Can't locate Env.pm" [ID 1925577.1]
How to change Hostname / IP for a Grid Infrastructure Oracle Restart Standalone Configuration (SIHA) [ID 1552810.1]

Related Posts
Changing The Cluster Name