Sunday, April 28, 2019

Add or Remove Database Options with chopt

chopt tool allows easier way of enabling or disabling certain database options after the database software installation. chopt does the binary relinking as part of enabling or disabling. Following shows an oracle database installation where real application testing is enabled (output generated using script here).
RAC             [    INSTALLED     ]
RAT             [    INSTALLED     ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]
chopt is used to disable RAT. Shutdown the database first and then use chopt.
srvctl stop database -db en18c

chopt disable rat

Writing to /opt/app/oracle/product/18.x.0/dbhome_1/install/disable_rat_2019-04-09_15-36-37pm.log...
/usr/bin/make -f /opt/app/oracle/product/18.x.0/dbhome_1/rdbms/lib/ins_rdbms.mk rat_off ORACLE_HOME=/opt/app/oracle/product/18.x.0/dbhome_1
/usr/bin/make -f /opt/app/oracle/product/18.x.0/dbhome_1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/opt/app/oracle/product/18.x.0/dbhome_1

srvctl start database -db en18c


Checking the output for enabled components after the change shows RAT is not installed any more.
RAC             [    INSTALLED     ]
RAT             [  NOT INSTALLED   ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]
Currently chopt could be used to enable or disable Oracle Advanced Analytics, Oracle OLAP, Oracle Partitioning and Oracle Real Application Testing options in 12.1 and above. For 11.2 the list include Oracle Data Mining RDBMS Files, Oracle Database Vault option, Oracle Label Security, Oracle OLAP, Oracle Partitioning and Oracle Real Application Testing.

Monday, April 22, 2019

Creating Gold Images on 18c

Both gridSetup.sh in grid infrastructure and runInstaller in Oracle database software has the option to create gold images from an installation. This facilitate standardization and uniformity. Moreover each time a patch is applied a new gold image could be created which will include the patched binaries. So any new home created from the gold image will also include the patch. This post shows steps for creating gold image for grid infrastructure and oracle database.

Creating Gold Image of Grid Infrastructure
Gold image from a grid infrastructure setup is created using gridSetup.sh with the createGoldImage option. By default this expect X-windows to be available. If not the execution will fail.
./gridSetup.sh -createGoldImage -destinationLocation /media/giGold
ERROR: Unable to verify the graphical display setup. This application requires X display. Make sure that xdpyinfo exist under PATH variable.

Secondly the destination location must be at least 23GB, even though the final zip file generated could be 5-6 GB (depending on excluding certain files). Execution will fail if there isn't sufficient space
./gridSetup.sh -createGoldImage -destinationLocation /media/giGold -silent
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-32707] The specified destination location (/media/giGold) does not have enough free space.
   ACTION: Provide a destination location path with at least (22,694) MB of free space.
If there's no risk of patches being rolled back, then the .patch_storage folder could be excluded from the gold image. This will reduce the size of the final zip file created. Other candidates for exclusions are files that have the server hostname in the name.
 ./gridSetup.sh -createGoldImage -destinationLocation /media/giGold -silent -exclFiles $GI_HOME/.patch_storage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /media/giGold/grid_home_2019-03-26_12-20-38PM.zip
Any new grid infrastructure installation done using the gold image will have the patches.
$ /opt/app/oracle/product/18.x.0/grid/OPatch/opatch lspatches
28864607;ACFS RELEASE UPDATE 18.5.0.0.0 (28864607)
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
The gold image doesn't zip all the files in the current installation. Some directories such as log directories are excluded automatically. In previous versions when installing from a gold image, these missing folders caused issue. But this appear to be fixed on 18c and no issues were encountered when a new installation was done using the gold image.




Creating Gold Image of Oracle Database
The gold image for Oracle database software is created using runInstaller. There are two runInstallers inside an OH. One is in $OH/runInstaller and other is in $OH/oui/bin/runInstaller. This gold image creating option is only available in $OH/runInstaller.
./runInstaller -createGoldImage -destinationLocation /media/ohGold -silent -exclFiles $ORACLE_HOME/.patch_storage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /media/ohGold/db_home_2019-03-26_03-35-48PM.zip
As mentioned earlier patch storage could be excluded from the gold image. Any new database installation created using the gold image will have the patches already applied.
/opt/app/oracle/product/18.x.0/dbhome_1/OPatch/opatch lspatches
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
Any new database created will also have the post patch scripts applied.
SQL> select PATCH_ID,PATCH_TYPE,ACTION,STATUS,SOURCE_VERSION,TARGET_VERSION,ACTION_TIME from dba_registry_sqlpatch;

  PATCH_ID PATCH_TYPE ACTION          STATUS                    SOURCE_VERSION  TARGET_VERSION  ACTION_TIME
---------- ---------- --------------- ------------------------- --------------- --------------- ------------------------------
  28822489 RU         APPLY           SUCCESS                   18.1.0.0.0      18.5.0.0.0      27-MAR-19 12.10.44.167872 PM
  27923415 INTERIM    APPLY           SUCCESS                   18.1.0.0.0      18.1.0.0.0      27-MAR-19 12.10.44.171197 PM

Update on 2021-02-15

It appears creating gold image excluding .patch_storage could cause issues down the line even if explicity patch rollback is not done. For example, a gold image was created from a GI home which had RU 19.5 and one-off patch 29013832 installed (needed to fix an issue on AFD). The installation using the gold image was fine until RU 19.10 was applied. During GI home patch the opatchauto command failed with and following could be seen on the patch logs
[Feb 15, 2021 3:25:56 PM] [INFO]    Patch 29013832:
                                    Copy Action: Directory "/opt/app/oracle/product/19.x.0/grid/.patch_storage/29013832_Oct_25_2019_13_06_10" does not exists or is not readable.
                                    'oracle.rdbms, 19.0.0.0.0': Cannot copy file from 'asmcmdafd.pm' to '/opt/app/oracle/product/19.x.0/grid/lib/asmcmdafd.pm'
[Feb 15, 2021 3:25:56 PM] [INFO]    Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
                                    The details are:

                                    Patch 29013832:
                                    Copy Action: Directory "/opt/app/oracle/product/19.x.0/grid/.patch_storage/29013832_Oct_25_2019_13_06_10" does not exists or is not readable.
                                    'oracle.rdbms, 19.0.0.0.0': Cannot copy file from 'asmcmdafd.pm' to '/opt/app/oracle/product/19.x.0/grid/lib/asmcmdafd.pm'
[Feb 15, 2021 3:25:56 PM] [SEVERE]  OUI-67073:UtilSession failed: Prerequisite check "CheckRollbackable" on auto-rollback patches failed.

It appears RU 19.10 does an implicity rollback of the patch 29013832 (as it is included in RU 19.10) but the old version of (19.5) is not available.
As old patch directory (29013832_Oct_25_2019_13_06_10) could not be copied over from an earlier setup, had to recover from this situation by reinstalling GI as software only and then configuring HAS.
So it maybe best to create the gold images with .patch_storage folder though that can add several GBs to the final size of the gold image.

Monday, April 15, 2019

DIA-49802: missing read, write, or execute permission on specified ADR home directory

Executing an 18c (18.5) Oracle restart reconfiguration after a hostname change resulted in following error.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-25-04AM.log
2019/03/27 08:25:06 CLSRSC-363: User ignored prerequisites during installation
Oracle Clusterware infrastructure error in OCRCONFIG (OS PID 8212): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

Oracle Clusterware infrastructure error in CLSCFG (OS PID 8223): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:26:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

2019/03/27 08:28:17 CLSRSC-214: Failed to start the resource 'ohasd'
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2019-03-27 08:26:28.832
[client(8848)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8848
2019-03-27 08:26:28.845
[client(8848)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:31.155
[client(8900)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8900
2019-03-27 08:26:31.168
[client(8900)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:33.140
[client(8950)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8950
2019-03-27 08:26:33.152
[client(8950)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:35.195
[client(9000)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9000
2019-03-27 08:26:35.208
[client(9000)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:37.231
[client(9050)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9050
2019-03-27 08:26:37.243
[client(9050)]CRS-2112:The OLR service started on node ip-172-31-1-113.

2019/03/27 08:28:17 CLSRSC-318: Failed to start Oracle OHASD service
Died at /opt/app/oracle/product/18.0.0/grid/crs/install/crsinstall.pm line 3226.

It appears during the reconfiguration, which is executed as root user the folders inside $ORACLE_BASE/diag/crs/`hostname -s`/crs gets created with root ownership but are written to as oracle (or grid if role separation is used) user.
cd /opt/app/oracle/diag/crs/ip-172-31-1-113/crs
[root@ip-172-31-1-113 crs]# ls -l

drwxrwx---. 2 root root  4096 Mar 27 08:24 alert
drwxrwx---. 2 root root  4096 Mar 27 08:24 cdump
drwxrwx---. 2 root root  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 root root  4096 Mar 27 08:24 lck
drwxrwx---. 4 root root  4096 Mar 27 08:29 log
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 root root  4096 Mar 27 08:24 stage
drwxrwx---. 2 root root  4096 Mar 27 08:24 sweep
drwxrwx---. 2 root root 12288 Mar 27 08:33 trace


This issue only happens during reconfiguration. On a new installation the ownership is set to the user installing the grid software, either oracle or grid. Changing the ownership to oracle (or grid) user in these directories resolved the issue.
chown -R oracle:oinstall *

[root@ip-172-31-1-113 crs]# ls -l
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 alert
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 cdump
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incident
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 lck
drwxrwx---. 4 oracle oinstall  4096 Mar 27 08:29 log
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 stage
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 sweep
drwxrwx---. 2 oracle oinstall 20480 Mar 27 08:41 trace
Re-run the Oracle restart configuration again.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-38-56AM.log
2019/03/27 08:38:59 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:39:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113'
CRS-2673: Attempting to stop 'ora.evmd' on 'ip-172-31-1-113'
CRS-2677: Stop of 'ora.evmd' on 'ip-172-31-1-113' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

ip-172-31-1-113     2019/03/27 08:42:19     /opt/app/oracle/product/18.0.0/grid/cdata/ip-172-31-1-113/backup_20190327_084219.olr     2532936542
2019/03/27 08:42:53 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Related Post
Changing Hostname in a Standalone DB Configuration with ASM