Wednesday, March 16, 2016

Opatch util Cleanup and .patch_storage

During a recent PSU apply it was noticed that grid home has grown to a considerable size.
du -sh grid4
16G     grid4
Out of the 16GB majority of space was consumed by the .patch_storage directory
du -sh grid4/.patch_storage
9.4G    grid4/.patch_storage
.patch_storage could be described loosely as the directory where opatch copies backup files during patch application. In case of rollback backup files from .patch_storage are copied to their original location.
In earlier versions (10g, 11gR1) optach util cleanup would remove older backup that are not needed for rollback and reduce the size of the .patch_storage. However in this case running this command didn't result in any reduction of the .patch_storage.
opatch util cleanup
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/app/11.2.0/grid4
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/11.2.0/grid4/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /opt/app/11.2.0/grid4/cfgtoollogs/opatch/opatch2016-03-09_15-20-16PM_1.log

Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y
Size of directory "/opt/app/11.2.0/grid4/.patch_storage" before cleanup is 10031648985 bytes.
Size of directory "/opt/app/11.2.0/grid4/.patch_storage" after cleanup is 10031648985 bytes.

UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.

OPatch succeeded.
Looking inside the .patch_storage it could be seen that some of the directories and files were created as a result of earlier PSU application far back as two years ago. The list below shows the size of the sub directories inside .patch_storage and the patch that resulted in creating this backup directory. Read 1641136.1 for GI PSU supplemental read which has all the patches released for GI 11.2.0.4
14M     17478514_Dec_6_2013_04_22_19  -  Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
33M     18031668_Feb_20_2014_05_15_58 -  Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
791M    18031731_Mar_17_2014_06_35_22 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.2 (18031731)"
555M    18031740_Mar_19_2014_09_06_37 -  Patch description:  "OCW Patch Set Update : 11.2.0.4.2 (18031740)"
28M     18522509_Jun_30_2014_08_14_42 -  Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
791M    18522514_May_6_2014_01_05_19  -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.3 (18522514)"
555M    18522515_Jun_16_2014_08_05_14 -  Patch description:  "OCW Patch Set Update : 11.2.0.4.3 (18522515)"
555M    19121549_Oct_6_2014_03_27_01  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.4 (19121549)"
30M     19121551_Oct_6_2014_10_07_57  -  Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
791M    19121552_Oct_6_2014_03_48_39  -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.4 (19121552)"
791M    19769469_Nov_13_2014_04_37_23 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.5 (19769469)"
555M    19769476_Dec_3_2014_02_08_05  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.5 (19769476)"
33M     19769489_Dec_28_2014_21_22_44 -  Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
1.7M    19852360_Oct_20_2014_08_17_43 -  Patch  19852360     : applied on Thu Jan 22 17:08:25 GMT 2015
40M     20299013_Mar_4_2015_02_27_44  -  Sub-patch  20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
791M    20299019_Mar_27_2015_15_26_30 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.6/7 (20299019)"
4.0M    20760982_Jun_4_2015_00_23_20  -  Sub-patch  20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
548M    20831122_Jul_1_2015_06_26_45  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.7 (20831122)"
796K    21352635_Sep_1_2015_07_49_44  -  Patch description:  "Database Patch Set Update : 11.2.0.4.8 (21352635)"
791M    21352642_Sep_3_2015_00_03_11  -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.8 (21352642)"
548M    21352649_Sep_2_2015_23_43_49  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.8 (21352649)"
17M     21948347_Dec_14_2015_03_31_48 -  Patch description:  "Database Patch Set Update : 11.2.0.4.160119 (21948347)" 
548M    21948348_Dec_13_2015_23_42_28 -  Patch description:  "OCW Patch Set Update : 11.2.0.4.160119 (21948348)"
791M    21948355_Nov_18_2015_00_55_35 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.160119 (21948355)"
From this it seems .patch_storage always accumulate new backups and never expire any. When asked Oracle via a SR it was confirmed that due to new composite patching mechanism introduced in 11.2, .patch_storage will keep on growing as some of the rollback files would have to come from earlier backups. Therefore it is not possible to delete any files in .patch_storage unless rollback is not expected (and even then it's not advised. read 403218.1).


It seems only time that opatch util Cleanup does any cleaning is during patch application itself. Following from the opatch log
[Mar 9, 2016 3:02:35 PM]     OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
                             You will be still able to rollback patches after this cleanup.
                             Do you want to proceed? [y|n]
[Mar 9, 2016 3:02:38 PM]     Y (auto-answered by -silent)
[Mar 9, 2016 3:02:38 PM]     User Responded with: Y
[Mar 9, 2016 3:02:39 PM]     Size of directory "/opt/app/11.2.0/grid4/.patch_storage" before cleanup is 10243859708 bytes.
[Mar 9, 2016 3:02:39 PM]     Deleting the directory "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/backup"
...
[Mar 9, 2016 3:02:39 PM]     Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/mode.txt"
[Mar 9, 2016 3:02:39 PM]     Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/make_cmds.txt"
[Mar 9, 2016 3:02:39 PM]     Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/remote_cmds_21948348.txt"
...
[Mar 9, 2016 3:02:40 PM]     Size of directory "/opt/app/11.2.0/grid4/.patch_storage" after cleanup is 9203930259 bytes.
[Mar 9, 2016 3:02:40 PM]     UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
                             deleted, Please refer log file.
In this case where the GI home has all the 11.2.0.4 PSU applied over a 2 year period has resulted in .patch_storage size of 10GB, on a 12.1.0.2 single instance environment applying GI PSU 12.1.0.2.2 - 12.1.0.2.160119 has resulted in a .patch_storage of 6.1G. So it's expected that throughout the life span of the database the .patch_storage (for both GI home and Oracle home) could grow larger than the minimum storage requirement listed for GI and Oracle home and installation locations must be sized with this growth in mind.

Useful metalink notes
Can You Delete $ORACLE_HOME/.patch_storage Directory ? [ID 403218.1]
How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space. [ID 550522.1]
Oracle Grid Infrastructure 11.2.0.4.x Patch Set Update SUPPLEMENTAL README [ID 1641136.1]

Tuesday, March 1, 2016

Changing Hostname in a Standalone DB Configuration with ASM

Grid infrastructure software is required to setup ASM in a standalone DB configuration. GI has hostname dependent directory paths and components. Changing the hostname in a such configuration requires re-configuring the GI software after the hostname has been changed. This post list the steps for changing the hostname in a standalone DB configuration with ASM.
The current hostname is rhel7.domain.net and this will changed to rhel7s.domain.net
In certain systems (in this case OS RHEL 7.0, kernel 3.10.0-123.el7.x86_64, Oracle 11.2.0.4) may run in to following error due to perl version related issues. Solution for this is to use the perl binary provided with GI.
$GI_HOME/crs/install/roothas.pl -deconfig -force
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
The MOS notes mention to change the hostname and then de-configure but this fails with the following error
export ORACLE_HOME=$GI_HOME
$GI_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Can't open /etc/oracle/scls_scr/rhel7s/grid/ohasdstr for write: No such file or directory at /opt/app/oracle/product/11.2.0/grid/crs/install/s_crsconfig_lib.pm line 1386.
Reason is that deconfig option is looking for a directory path that does not exists
# ls /etc/oracle/scls_scr/
rhel7
It maybe possible to rename this directory and run the deconfig. But instead the deconfig was run (as root) without changing the hostname (reverting to original hostname).
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
CRS resources for listeners are still configured
PRKO-2573 : ONS daemon is already stopped.
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel7'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel7'
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel7'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel7' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel7' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel7' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel7'
CRS-2677: Stop of 'ora.asm' on 'rhel7' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel7'
CRS-2677: Stop of 'ora.cssd' on 'rhel7' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7'
CRS-2677: Stop of 'ora.evmd' on 'rhel7' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack
Once the deconfig is complete change the hostname
hostname
rhel7s.domain.net
Run (as root) the roothas.pl script to configure the HAS again.
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel7s successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

rhel7s     2016/02/12 13:10:13     /opt/app/oracle/product/11.2.0/grid/cdata/rhel7s/backup_20160212_131013.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
At this stage the services would be in the following states
Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.cssd                            ora.cssd.type                  OFFLINE            OFFLINE
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
As grid user, enable the cssd auto start and stop and start the HAS again
$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
$ crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7s'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7s'
CRS-2677: Stop of 'ora.evmd' on 'rhel7s' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7s' has completed
CRS-4133: Oracle High Availability Services has been stopped.
$ crsctl start has
CRS-4123: Oracle High Availability Services has been started.
Check the states of the services and HAS service
Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE

$ crsctl check has
CRS-4638: Oracle High Availability Services is online
GI has its services but listener, ASM and DB are missing. These need to be manually added to the configuration. First add and start the listener
$ srvctl add listener
$ srvctl start listener

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE


Add the ASM and set auto start option to 1. In this case the ASM Spfile was not residing in a ASM diskgroup.
$ srvctl add asm
$ crsctl modify resource "ora.asm" -attr "AUTO_START=1"
$ srvctl modify asm -p /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora -l listener

$ srvctl config asm
ASM home: /opt/app/oracle/product/11.2.0/grid
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

$ crsctl stop has
$ crsctl start has

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
Finally add the database and start it
$ srvctl add database -d std11g2 -o $ORACLE_HOME -p +DATA/std11g2/spfilestd11g2.ora
$ srvctl config database -d std11g2
Database unique name: std11g2
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/std11g2/spfilestd11g2.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: std11g2
Disk Groups:
Services:
$ srvctl start database -d std11g2

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type              ONLINE             ONLINE             rhel7s
This conclude the changing of hostname in a standalone DB configuration with ASM.

Useful metalink notes
How to Reconfigure Oracle Restart on 12c / 12.1 [ID 1570358.1]
rootcrs.pl/roothas.pl Fails With Can't locate Env.pm [ID 2019784.1]
rootcrs.pl/roothas.pl Fails With "Can't locate Env.pm" [ID 1925577.1]
How to change Hostname / IP for a Grid Infrastructure Oracle Restart Standalone Configuration (SIHA) [ID 1552810.1]

Related Posts
Changing The Cluster Name