Monday, April 18, 2016

Valid Node Checking For Registration (VNCR)

Valid Node Checking for Registration (VNCR) allows restriction of instance registration to come only from know servers. This has replaced the use of COST (setting up COST for 11gR1 SE and 11gR2 SE) for protecting against TNS Listener Poison Attack. COST could be still used if it is used for reasons other than listener restriction (for more read 1340831.1).
VNCR is available on 11.2.0.4 and 12c so if the DB is 11.2.0.3 then COST is still the only option available against this attack. Below is an example of setting VNCR on a single instance database. For RAC with SCAN listeners refer the MOS listed at the end of the post.
The listener runs on server with IP 192.168.0.66 (hpc1.domain.net). Currently no DB is registered.
lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:06:44

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                22-FEB-2016 13:06:41
Uptime                    0 days 0 hr. 0 min. 3 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
The listener supports no services
The command completed successfully
The DB (something to register with this listener) runs on server with IP 192.168.0.99. To register with the listener on 192.168.0.66 add a remote listener entry on tnsnames.ora and register the database
cat tnsnames.ora
listener_name =
  (DESCRIPTION=
    (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.66)(PORT=1521))
  )

SQL> alter system set remote_listener='listener_name' scope=both;
System altered.

SQL> alter system register;
System altered.
Check the listener status
lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:08:51

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                22-FEB-2016 13:06:41
Uptime                    0 days 0 hr. 2 min. 10 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
Services Summary...
Service "fgacdb" has 1 instance(s).
  Instance "fgacdb", status READY, has 1 handler(s) for this service...
Service "fgacdbXDB" has 1 instance(s).
  Instance "fgacdb", status READY, has 1 handler(s) for this service...
The command completed successfully
Above shows that remote database has registered with the listener.



In order to prevent this type of remote registrations enable valid node checking on the listener. To enable VNCR on 11.2.0.4 add following entry to listener.ora file In this case the listener name is "LISTENER".
VALID_NODE_CHECKING_REGISTRATION_LISTENER=1
Reload the listener and check the status. The remote listener registration is blocked.
$ lsnrctl reload
$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:24:17

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                22-FEB-2016 13:06:41
Uptime                    0 days 0 hr. 17 min. 35 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
The listener supports no services
Any attempt to register with the listener is blocked and the listener.log will have an entry similar to following
Listener(VNCR option 1) rejected Registration request from destination 192.168.0.99
22-FEB-2016 13:31:47 * service_register_NSGR * 1182
TNS-01182: Listener rejected registration of service ""
On 12c is VNCR is enabled by default and any attempt for remote registration is rejected and listener log will have an entry as above.

Useful metalink notes
Valid Node Checking For Registration (VNCR) [ID 1600630.1]
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]

Related Posts
Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

Friday, April 1, 2016

Upgrading RHEL 7 OS in a 11gR2 (11.2.0.4) and 12cR1 (12.1.0.2) Oracle Restart Environments

As per Oracle documentation "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade." There are two earlier posts which shows upgrading of RHEL5 and RHEL6 OS in a 11gR2 RAC environment. This post shows relinking of GI and Oracle homes after upgrading of RHEL 7 in 11.2.0.4 and 12.1.0.2 single instance (Oracle restart) environments.
1. The current RHEL version is 7.1 and the kernel is
uname -r
3.10.0-123.el7.x86_64
2. Before the OS upgrade stop the HAS and disable the auto start on reboot.
crsctl stop has
crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
3. Comment the entries in oracle-ohasd.service
cat /etc/systemd/system/oracle-ohasd.service
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

[Unit]
#Description=Oracle High Availability Services
#After=syslog.target

[Service]
#ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
#Type=simple
#Restart=always

[Install]
#WantedBy=multi-user.target graphical.target
4. Upgrade the RHEL 7 OS. Kernel and RHEL7 version after the upgrade are
uname -r
3.10.0-327.el7.x86_64
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
5. Uncomment the entires in oracle-ohasd.service file and enable auto start of HAS
crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.
Until this point the steps are common for both 11.2.0.4 and 12.1.0.2.

Relinking 11.2.0.4 GI and Oracle Homes
1.Relinking Oracle Home as oracle user. Make sure the relink binary in the PATH is from Oracle home
which relink
/opt/app/oracle/product/11.2.0/dbhome_4/bin/relink
Run relink all
cd $ORACLE_HOME/bin
./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_4/install/relink.log
At the end of the relink the log file have the following
test ! -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_4/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
2. Relinkning GI Home
Set ORACLE_HOME=$GI_HOME
As root unlock the GI Home. Using the RHEL 7 provided perl version will result in following error
# perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
Solution for this was mentioned on the changing of hostname post which is to use oracle provided perl
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Successfully unlock /opt/app/oracle/product/11.2.0/grid
3. As grid user run the relink
$ORACLE_HOME/bin/relink
writing relink log to: /opt/app/oracle/product/11.2.0/grid/install/relink.log
The log file will have the following entry at the end
tail /opt/app/oracle/product/11.2.0/grid/install/relink.log

log tail
test ! -f /opt/app/oracle/product/11.2.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/grid/bin/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracleO
mv /opt/app/oracle/product/11.2.0/grid/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/grid/bin/oracle
4. As root patch the GI Home
# cd $ORACLE_HOME/rdbms/install
# ./rootadd_rdbms.sh

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -patch
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Broadcast message from systemd-journald@rhel7s.domain.net (Thu 2016-02-18 16:50:14 GMT):
dracut[16456]: dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed

Message from syslogd@rhel7s at Feb 18 16:50:14 ...
 dracut:dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed
ACFS driver install actions failed
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for ohasd to start.
During the patching the HAS stack start will fail, this is expected.
5. Reboot the host after uncommenting the lines in oracle-ohasd.service together with enabling HAS service (crsctl enable has). After the reboot HAS stack will be up and running as before
Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type              ONLINE             ONLINE             rhel7s


Relinking 12.1.0.2 GI and Oracle Homes
1. Relinking Oracle home as oracle user. Make sure the relink binary in PATH is from Oracle home. Run relink all to relink the oracle home.
relink all
writing relink log to: /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
The log file end with the following
tail /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
test ! -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracleO
mv /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
2. As root unlock the GI Home by using the $GI_HOME/crs/install/roothas.sh. Unlike 11.2.0.4 there are no issues in unlocking GI home in 12c
./roothas.sh -unlock
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
2016/02/19 18:52:17 CLSRSC-347: Successfully unlock /opt/app/oracle/product/12.1.0/grid
3. As grid user run the relink.
which relink
/opt/app/oracle/product/12.1.0/grid/bin/relink
$ relink
writing relink log to: /opt/app/oracle/product/12.1.0/grid/install/relink.log
The log file will end with the following
tail /opt/app/oracle/product/12.1.0/grid/install/relink.log

test ! -f /opt/app/oracle/product/12.1.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/grid/bin/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracleO
mv /opt/app/oracle/product/12.1.0/grid/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/grid/bin/oracle
4. As root patch the GI Home
# $GI_HOME/rdbms/install/rootadd_rdbms.sh

./roothas.sh -patch
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
2016/02/19 18:59:50 CLSRSC-199: Timed out waiting for OHASD to start

The command '/opt/app/oracle/product/12.1.0/grid/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid/perl/lib -I/opt/app/oracle/product/12.1.0/grid/crs/install /opt/app/oracle/product/12.1.0/grid/crs/install/roothas.pl -patch' execution failed
5. Similar to 11.2.0.4, start of the HAS stack will fail, this is expected. Reboot the host after uncommenting the lines in oracle-ohasd.service together with enabling HAS service (crsctl enable has). After the reboot HAS will be up and running.
Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             rhel7
ora.asm                   ora.asm.type              ONLINE             ONLINE             rhel7
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             rhel7
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             rhel7
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.se2db.db              ora.database.type         ONLINE             ONLINE             rhel7
This concludes the steps for relinking GI and Oracle homes after RHEL 7 OS upgrade.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment
Upgrading ASMLib and OS in 11gR1 RAC Environment

Update on 2017-07-10
As per 2282371.1 and 2281492.1 certain RHEL 7 kernel versions (3.10.0-514.21.2.el7.x86_64) cause issues with regard to starting of OHASD in 11.2.0.4. Avoid these kernel versions when installing or upgrading OS when 11.2.0.4 is installed.

Useful metalink notes
OHASD Fails to Start With Kernel Version 3.10.0-514.21.2.el7.x86_64 [ID 2281492.1]
Grid Infrastructure Fails to Start OHASD With RedHat Linux or Oracle Linux with RedHat Compatible Kernel (RHCK) Version 3.10.0-514.21.2.EL7.X86_64 or Higher [ID 2282371.1]

Wednesday, March 16, 2016

Opatch util Cleanup and .patch_storage

During a recent PSU apply it was noticed that grid home has grown to a considerable size.
du -sh grid4
16G     grid4
Out of the 16GB majority of space was consumed by the .patch_storage directory
du -sh grid4/.patch_storage
9.4G    grid4/.patch_storage
.patch_storage could be described loosely as the directory where opatch copies backup files during patch application. In case of rollback backup files from .patch_storage are copied to their original location.
In earlier versions (10g, 11gR1) optach util cleanup would remove older backup that are not needed for rollback and reduce the size of the .patch_storage. However in this case running this command didn't result in any reduction of the .patch_storage.
opatch util cleanup
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/app/11.2.0/grid4
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/11.2.0/grid4/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /opt/app/11.2.0/grid4/cfgtoollogs/opatch/opatch2016-03-09_15-20-16PM_1.log

Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y
Size of directory "/opt/app/11.2.0/grid4/.patch_storage" before cleanup is 10031648985 bytes.
Size of directory "/opt/app/11.2.0/grid4/.patch_storage" after cleanup is 10031648985 bytes.

UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.

OPatch succeeded.
Looking inside the .patch_storage it could be seen that some of the directories and files were created as a result of earlier PSU application far back as two years ago. The list below shows the size of the sub directories inside .patch_storage and the patch that resulted in creating this backup directory. Read 1641136.1 for GI PSU supplemental read which has all the patches released for GI 11.2.0.4
14M     17478514_Dec_6_2013_04_22_19  -  Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
33M     18031668_Feb_20_2014_05_15_58 -  Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
791M    18031731_Mar_17_2014_06_35_22 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.2 (18031731)"
555M    18031740_Mar_19_2014_09_06_37 -  Patch description:  "OCW Patch Set Update : 11.2.0.4.2 (18031740)"
28M     18522509_Jun_30_2014_08_14_42 -  Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
791M    18522514_May_6_2014_01_05_19  -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.3 (18522514)"
555M    18522515_Jun_16_2014_08_05_14 -  Patch description:  "OCW Patch Set Update : 11.2.0.4.3 (18522515)"
555M    19121549_Oct_6_2014_03_27_01  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.4 (19121549)"
30M     19121551_Oct_6_2014_10_07_57  -  Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
791M    19121552_Oct_6_2014_03_48_39  -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.4 (19121552)"
791M    19769469_Nov_13_2014_04_37_23 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.5 (19769469)"
555M    19769476_Dec_3_2014_02_08_05  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.5 (19769476)"
33M     19769489_Dec_28_2014_21_22_44 -  Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
1.7M    19852360_Oct_20_2014_08_17_43 -  Patch  19852360     : applied on Thu Jan 22 17:08:25 GMT 2015
40M     20299013_Mar_4_2015_02_27_44  -  Sub-patch  20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
791M    20299019_Mar_27_2015_15_26_30 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.6/7 (20299019)"
4.0M    20760982_Jun_4_2015_00_23_20  -  Sub-patch  20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
548M    20831122_Jul_1_2015_06_26_45  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.7 (20831122)"
796K    21352635_Sep_1_2015_07_49_44  -  Patch description:  "Database Patch Set Update : 11.2.0.4.8 (21352635)"
791M    21352642_Sep_3_2015_00_03_11  -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.8 (21352642)"
548M    21352649_Sep_2_2015_23_43_49  -  Patch description:  "OCW Patch Set Update : 11.2.0.4.8 (21352649)"
17M     21948347_Dec_14_2015_03_31_48 -  Patch description:  "Database Patch Set Update : 11.2.0.4.160119 (21948347)" 
548M    21948348_Dec_13_2015_23_42_28 -  Patch description:  "OCW Patch Set Update : 11.2.0.4.160119 (21948348)"
791M    21948355_Nov_18_2015_00_55_35 -  Patch description:  "ACFS Patch Set Update : 11.2.0.4.160119 (21948355)"
From this it seems .patch_storage always accumulate new backups and never expire any. When asked Oracle via a SR it was confirmed that due to new composite patching mechanism introduced in 11.2, .patch_storage will keep on growing as some of the rollback files would have to come from earlier backups. Therefore it is not possible to delete any files in .patch_storage unless rollback is not expected (and even then it's not advised. read 403218.1).


It seems only time that opatch util Cleanup does any cleaning is during patch application itself. Following from the opatch log
[Mar 9, 2016 3:02:35 PM]     OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
                             You will be still able to rollback patches after this cleanup.
                             Do you want to proceed? [y|n]
[Mar 9, 2016 3:02:38 PM]     Y (auto-answered by -silent)
[Mar 9, 2016 3:02:38 PM]     User Responded with: Y
[Mar 9, 2016 3:02:39 PM]     Size of directory "/opt/app/11.2.0/grid4/.patch_storage" before cleanup is 10243859708 bytes.
[Mar 9, 2016 3:02:39 PM]     Deleting the directory "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/backup"
...
[Mar 9, 2016 3:02:39 PM]     Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/mode.txt"
[Mar 9, 2016 3:02:39 PM]     Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/make_cmds.txt"
[Mar 9, 2016 3:02:39 PM]     Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/remote_cmds_21948348.txt"
...
[Mar 9, 2016 3:02:40 PM]     Size of directory "/opt/app/11.2.0/grid4/.patch_storage" after cleanup is 9203930259 bytes.
[Mar 9, 2016 3:02:40 PM]     UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
                             deleted, Please refer log file.
In this case where the GI home has all the 11.2.0.4 PSU applied over a 2 year period has resulted in .patch_storage size of 10GB, on a 12.1.0.2 single instance environment applying GI PSU 12.1.0.2.2 - 12.1.0.2.160119 has resulted in a .patch_storage of 6.1G. So it's expected that throughout the life span of the database the .patch_storage (for both GI home and Oracle home) could grow larger than the minimum storage requirement listed for GI and Oracle home and installation locations must be sized with this growth in mind.

Useful metalink notes
Can You Delete $ORACLE_HOME/.patch_storage Directory ? [ID 403218.1]
How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space. [ID 550522.1]
Oracle Grid Infrastructure 11.2.0.4.x Patch Set Update SUPPLEMENTAL README [ID 1641136.1]