Monday, April 15, 2019

DIA-49802: missing read, write, or execute permission on specified ADR home directory

Executing an 18c (18.5) Oracle restart reconfiguration after a hostname change resulted in following error.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-25-04AM.log
2019/03/27 08:25:06 CLSRSC-363: User ignored prerequisites during installation
Oracle Clusterware infrastructure error in OCRCONFIG (OS PID 8212): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

Oracle Clusterware infrastructure error in CLSCFG (OS PID 8223): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:26:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

2019/03/27 08:28:17 CLSRSC-214: Failed to start the resource 'ohasd'
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2019-03-27 08:26:28.832
[client(8848)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8848
2019-03-27 08:26:28.845
[client(8848)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:31.155
[client(8900)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8900
2019-03-27 08:26:31.168
[client(8900)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:33.140
[client(8950)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8950
2019-03-27 08:26:33.152
[client(8950)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:35.195
[client(9000)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9000
2019-03-27 08:26:35.208
[client(9000)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:37.231
[client(9050)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9050
2019-03-27 08:26:37.243
[client(9050)]CRS-2112:The OLR service started on node ip-172-31-1-113.

2019/03/27 08:28:17 CLSRSC-318: Failed to start Oracle OHASD service
Died at /opt/app/oracle/product/18.0.0/grid/crs/install/crsinstall.pm line 3226.

It appears during the reconfiguration, which is executed as root user the folders inside $ORACLE_BASE/diag/crs/`hostname -s`/crs gets created with root ownership but are written to as oracle (or grid if role separation is used) user.
cd /opt/app/oracle/diag/crs/ip-172-31-1-113/crs
[root@ip-172-31-1-113 crs]# ls -l

drwxrwx---. 2 root root  4096 Mar 27 08:24 alert
drwxrwx---. 2 root root  4096 Mar 27 08:24 cdump
drwxrwx---. 2 root root  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 root root  4096 Mar 27 08:24 lck
drwxrwx---. 4 root root  4096 Mar 27 08:29 log
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 root root  4096 Mar 27 08:24 stage
drwxrwx---. 2 root root  4096 Mar 27 08:24 sweep
drwxrwx---. 2 root root 12288 Mar 27 08:33 trace


This issue only happens during reconfiguration. On a new installation the ownership is set to the user installing the grid software, either oracle or grid. Changing the ownership to oracle (or grid) user in these directories resolved the issue.
chown -R oracle:oinstall *

[root@ip-172-31-1-113 crs]# ls -l
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 alert
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 cdump
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incident
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 lck
drwxrwx---. 4 oracle oinstall  4096 Mar 27 08:29 log
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 stage
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 sweep
drwxrwx---. 2 oracle oinstall 20480 Mar 27 08:41 trace
Re-run the Oracle restart configuration again.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-38-56AM.log
2019/03/27 08:38:59 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:39:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113'
CRS-2673: Attempting to stop 'ora.evmd' on 'ip-172-31-1-113'
CRS-2677: Stop of 'ora.evmd' on 'ip-172-31-1-113' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

ip-172-31-1-113     2019/03/27 08:42:19     /opt/app/oracle/product/18.0.0/grid/cdata/ip-172-31-1-113/backup_20190327_084219.olr     2532936542
2019/03/27 08:42:53 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Related Post
Changing Hostname in a Standalone DB Configuration with ASM