Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware

Installing 12cR1 RAC without the use of the new features such as the Oracle Flex ASM is pretty much same as installing 11gR2 RAC with role separation. This post is not a step by step guide to installing 12cR1 RAC but list the main differences in the installation process compared to previous versions and pitfalls to look out for.
Among the noticeable changes and new features 12c doesn't support storing clusterware files in block devices as such provisions must be made either to store in the shared file systems or in ASM. Even if upgrading ocr files must be moved to ASM before the upgrade.
12c supports IPv6 for public IP and VIP. However this post uses IPv4 for public and VIP.
12c has also introduced several new administrative privileges for job role separation for database related tasks in data guard, key management and backup.
1. The RHEL 6 version used for the setup is RHEL 6 - U4

# uname -r
2.6.32-358.el6.x86_64

2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.

groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid

grid user was made part of the dba group as per 1084186.1. (If this issue is still there on 12c was not tested).
3. Create user equivalence manually and do not wait for OUI to do this during the installation even though Oracle documentation says "You can configure SSH from the OUI interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.". Without this manual user equivalence none of the cluvfy commands will complete successfully.
4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission. For GI_HOME

mkdir -p /opt/app/12.1.0/grid 
chown -R grid:oinstall /opt/app
chown grid:oinstall /opt
chmod -R 775 /opt/app
chmod 775 /opt

For ORACLE_BASE

mkdir -p /opt/app/oracle
chown oracle:oinstall /opt/app/oracle
chmod 775 /opt/app/oracle

After running the root.sh the ownership of these directories (for GI_HOME) will be changed to root. But before the installation it must be owned by grid with oinstall as group and both user and group must have read,write and execute permissions.
5. Verify pre-req package list is installed. Follow GI installation guide and metalink note Requirements for Installing Oracle Database 12.1 on RHEL6 or OL6 64-bit (x86-64) [ID 1529864.1] for list of rpms required. Starting with 11.2.0.2 and including 12c "the 32-bit packages are required only for 32-bit client installs". In this setup servers used had both 32-bit and 64-bit version were installed.
6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768

7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file

Host *
 ForwardX11 no

8. Disable tty on .bashrc for both grid and oracle user

if [ -t 0 ]; then
 stty intr ^C
fi

and also set umask value in .bashrc for grid and oracle user

umask 022

9. ASMLib is not used for this RAC setup. Instead block devices are used as ASM disk. Setup udev rules for block devices with following ownerships and permissions.

KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

If scsi_id command is used to find the UUID then the command listed on the GI install documentation fails on RHEL 6.

# /sbin/scsi_id -g -s /block/sdb/sdb1
/sbin/scsi_id: invalid option -- 's'

-s is not a supported option on RHEL 6. Instead use the following

/sbin/scsi_id -g -u -d /dev/sdb

Further more Oracle documentation mentions with external redundancy a minimum of 1 disk of 700MB size is sufficient to store both OCR and vote disk when not configuring cluster management repository. However it must be stated 700MB is not sufficient the disk space should be at least 1069MB.

In practice it is best to use normal or high redundancy on the disk group used to store OCR and vote disk.
10. Running cluvfy fails with the following message

ERROR:
Reference data is not available for verifying prerequisites on this operating system distribution
Verification cannot proceed

As mentioned in upgrade to 12c post install redhat-release-6Server-1.noarch.rpm found in the pack available with 1514012.1. (Also refer 1567127.1). After the aforementioned rpm is installed cluvfy can continue without any issue. If the cluvfy or OUI cannot find the UUID for a disk during the disk share check following message will be shown. From cluvfy

Checking Devices for ASM...

ERROR:  /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
        rhel12c2,rhel12c1

Checking for shared devices...

PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes

From OUI

Error is due to test system using VirtualBox to setup the RAC and partitions not returning an UUID. Installation could be continued ignoring this error. In a proper system where UUID is available the cluvfy would have the following messages when these check succeed.

Checking Devices for ASM...

Checking for shared devices...

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/sdb1                              Disk


Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/sdb1" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/sdb1" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/sdb1" PASSED

Checking consistency of device size across all nodes...
Consistency check of device size for "/dev/sdb1" PASSED

Below is the output from running pre crsinst.

[grid@rhel12c1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel12c1,rhel12c2 -asm -asmgrp asmadmin -presence local -asmdev /dev/sdb1 -crshome /opt/app/12.1.0/grid -fixup  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel12c1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel12c1                              yes
  rhel12c2                              yes
Result: Node reachability check passed from node "rhel12c1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed

Verification of the hosts config file successful


Interface information for node "rhel12c1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:CB:D8:AE 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:92:0B:69 1500


Interface information for node "rhel12c2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:9C:66:81 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:1E:0E:A8 1500


Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.0.93]          rhel12c2[192.168.0.94]          yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.0.93           rhel12c2:192.168.0.94           passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.1.87]          rhel12c2[192.168.1.88]          yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.1.87           rhel12c2:192.168.1.88           passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel12c1 eth0:192.168.0.93
rhel12c2 eth0:192.168.0.94

Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel12c1 eth1:192.168.1.87
rhel12c2 eth1:192.168.1.88
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.4199GB (4634568.0KB)    4GB (4194304.0KB)         passed
  rhel12c1      4.4199GB (4634568.0KB)    4GB (4194304.0KB)         passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.2905GB (4498884.0KB)    50MB (51200.0KB)          passed
  rhel12c1      4.1755GB (4378320.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
  rhel12c1      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel12c2      /             21.3525GB     7.9635GB      passed
  /var              rhel12c2      /             21.3525GB     7.9635GB      passed
  /etc              rhel12c2      /             21.3525GB     7.9635GB      passed
  /opt/app/12.1.0/grid  rhel12c2      /             21.3525GB     7.9635GB      passed
  /sbin             rhel12c2      /             21.3525GB     7.9635GB      passed
  /tmp              rhel12c2      /             21.3525GB     7.9635GB      passed
Result: Free disk space check passed for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"

Check: Free disk space for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel12c1      /             18.3523GB     7.9635GB      passed
  /var              rhel12c1      /             18.3523GB     7.9635GB      passed
  /etc              rhel12c1      /             18.3523GB     7.9635GB      passed
  /opt/app/12.1.0/grid  rhel12c1      /             18.3523GB     7.9635GB      passed
  /sbin             rhel12c1      /             18.3523GB     7.9635GB      passed
  /tmp              rhel12c1      /             18.3523GB     7.9635GB      passed
Result: Free disk space check passed for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists(501)
  rhel12c1      passed                    exists(501)

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "dba"

Check: Group existence for "asmadmin"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "asmadmin"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c2          yes           yes           yes           yes           passed
  rhel12c1          yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Membership of user "grid" in group "asmadmin"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "grid" in group "asmadmin" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      3                         3,5                       passed
  rhel12c1      3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          65536         65536         passed
  rhel12c1          hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          1024          1024          passed
  rhel12c1          soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          16384         16384         passed
  rhel12c1          hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          2047          2047          passed
  rhel12c1          soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      x86_64                    x86_64                    passed
  rhel12c1      x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      2.6.32-358.el6.x86_64     2.6.32                    passed
  rhel12c1      2.6.32-358.el6.x86_64     2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          250           passed
  rhel12c2          3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          385280        385280        32000         passed
  rhel12c2          385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          100           passed
  rhel12c2          3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          128           128           128           passed
  rhel12c2          128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          68719476736   68719476736   2372898816    passed
  rhel12c2          68719476736   68719476736   2372898816    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4096          4096          4096          passed
  rhel12c2          4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4294967296    4294967296    463456        passed
  rhel12c2          4294967296    4294967296    463456        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          6815744       6815744       6815744       passed
  rhel12c2          6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel12c2          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       262144        passed
  rhel12c2          4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       4194304       passed
  rhel12c2          4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          1048576       1048576       262144        passed
  rhel12c2          1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          2097152       2097152       1048576       passed
  rhel12c2          2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3145728       3145728       1048576       passed
  rhel12c2          3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rhel12c1      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel12c1      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel12c1      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
  rhel12c1      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
  rhel12c1      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rhel12c1      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rhel12c1      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
  rhel12c1      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
  rhel12c1      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      ksh-20100621-19.el6       ksh-...                   passed
  rhel12c1      ksh-20100621-19.el6       ksh-...                   passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      make-3.81-20.el6          make-3.81                 passed
  rhel12c1      make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
  rhel12c1      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
  rhel12c1      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel12c1      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel12c1      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
  rhel12c1      nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
Result: Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          6200          TCP           yes           successful
  rhel12c1          6200          TCP           yes           successful
  rhel12c2          6100          TCP           yes           successful
  rhel12c1          6100          TCP           yes           successful
Result: Port availability check passed for ports "6200,6100"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed

Check for consistency of root user's primary group passed

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
  rhel12c1      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
Result: Package existence check passed for "cvuqdisk"


Checking Devices for ASM...

ERROR:  /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
        rhel12c2,rhel12c1

Checking for shared devices...

PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes

UDev attributes check for ASM Disks started...
Checking udev settings for device "/dev/sdb1"
  Device            Owner         Group         Permissions   Result
  ----------------  ------------  ------------  ------------  ----------------
  sdb1              grid          asmadmin      0660          passed

  sdb1              grid          asmadmin      0660          passed

Result: UDev attributes check passed for ASM Disks

Result: Devices check for ASM passed


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    does not exist
  rhel12c1      passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      0022                      0022                      passed
  rhel12c1      0022                      0022                      passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" passed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel12c2      no                        passed
  rhel12c1      no                        passed
Daemon not configured check passed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel12c2      no                        passed
  rhel12c1      no                        passed
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed


NOTE:
No fixable verification failures to fix

Pre-check for cluster services setup was successful.

Despite the issues on the ASM disk share cluvfy reports pre-reqs to be successful.



11. Start the clusterware installation by executing runInstaller.

12c provides many "use for" options for NICs. Since this is not setting up Flex ASM for eth1 select private. Oracle documentation states "Oracle Flex ASM can use either the same private networks as Oracle Clusterware, or use its own dedicated private networks. Each network can be classified PUBLIC, ASM & PRIVATE, PRIVATE, or ASM."

Selecting no prompts the below shown message. If GI management repository is installed then ASM space requirement would be higher. Refer Oracle documentation and 12c upgrade post for space requirements for management repository

12. When prompted execute the root.sh scripts. Below is the output from running the root.sh on first node.

[root@rhel12c1 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:35:28 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2013/08/20 17:36:29 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group clusterdg created successfully.

CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c09b02f830b64f49bf06b75f49b272ee.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c09b02f830b64f49bf06b75f49b272ee (/dev/sdb1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c1'
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:44:10 CLSRSC-343: Successfully started Oracle clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
2013/08/20 17:45:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Running root.sh on second node.

[root@rhel12c2 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:45:54 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2013/08/20 17:46:24 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c2'
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:52:08 CLSRSC-343: Successfully started Oracle clusterware stack

2013/08/20 17:52:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

13. Once root.sh is completed click OK on the execute script dialog to continue with the remaining configurations after which clusterware installation is complete.
14. Verify the cluster installation with cluvfy post crsinst

cluvfy stage -post crsinst -n rhel12c1,rhel12c2 -verbose

15. Use ASMCA to create additional ASM disk groups that will be used for data and flash recovery when database is setup. Cluvfy could be used to verify if the disk are shared

cluvfy comp ssa -n rhel12c1,rhel12c2 -s /dev/sdc1,/dev/sdd1 -t data -r 12.1 -verbose

Next step is to install the Oracle Database software.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Converting 11gR2 RAC from Standard Edition to Enterprise Edition

The existing system is a 11gR2 (11.2.0.3.7) Standard Edition (SE) RAC with role separation on RHEL 6. The blog post lists the steps for converting the SE RAC to Enterprise Edition (EE) RAC. The SE home is dbhome_1 and EE home is dbhome_2.
1.Take a full backup of the database as precaution in case have to revert back to the standard edition setup again.
2. Since the database home will be updated in the OCR take a backup of the OCR and also take copies of backup files created automatically from all cluster nodes.

[root@rhel6m1 grid]# ocrconfig -manualbackup
rhel6m2     2013/08/14 14:57:27     /opt/app/11.2.0/grid/cdata/rhel6m-cluster/backup_20130814_145727.ocr

3. Get the status of registry components, so validity of these components could be compared before and after the conversion. There are no invalid components before the conversion.

SQL> select comp_id,comp_name,version,status,schema from dba_registry;

COMP_ID  COMP_NAME                                VERSION                        STATUS SCHEMA
-------- ---------------------------------------- ------------------------------ ------ ------
XDB      Oracle XML Database                      11.2.0.3.0                     VALID  XDB
OWM      Oracle Workspace Manager                 11.2.0.3.0                     VALID  WMSYS
EXF      Oracle Expression Filter                 11.2.0.3.0                     VALID  EXFSYS
RUL      Oracle Rules Manager                     11.2.0.3.0                     VALID  EXFSYS
CATALOG  Oracle Database Catalog Views            11.2.0.3.0                     VALID  SYS
CATPROC  Oracle Database Packages and Types       11.2.0.3.0                     VALID  SYS
JAVAVM   JServer JAVA Virtual Machine             11.2.0.3.0                     VALID  SYS
XML      Oracle XDK                               11.2.0.3.0                     VALID  SYS
CATJAVA  Oracle Database Java Packages            11.2.0.3.0                     VALID  SYS
RAC      Oracle Real Application Clusters         11.2.0.3.0                     VALID  SYS

10 rows selected.

4. Find out the patch levels on SE Oracle Home and GI Home. Once the EE is installed and before moving the DB to EE home it must be patched to the same level as the SE Home and not to a higher level than the GI home. The current patches on SE home are

[oracle@rhel6m1 ~]$ /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -local | grep Patch
Oracle Interim Patch Installer version 11.2.0.3.4
OPatch version    : 11.2.0.3.4
Patch  16619892     : applied on Wed Jul 31 13:43:23 BST 2013
Unique Patch ID:  16346737
Patch description:  "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch  16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  16619898     : applied on Wed Jul 31 13:42:07 BST 2013
Unique Patch ID:  16376391
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.7 (16742216)"
OPatch succeeded.

Patches on GI home are

[grid@rhel6m1 ~]$ /opt/app/11.2.0/grid/OPatch/opatch lsinventory -local | grep Patch
Oracle Interim Patch Installer version 11.2.0.3.4
OPatch version    : 11.2.0.3.4
Patch  16619892     : applied on Wed Jul 31 13:54:30 BST 2013
Unique Patch ID:  16346737
Patch description:  "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch  16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  16619898     : applied on Wed Jul 31 13:52:27 BST 2013
Unique Patch ID:  16376391
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.7 (16742216)"
OPatch succeeded.

5. Install the EE across the cluster.
Select install software only option.

Select all the nodes where SE was installed

Select Enterprise Edition

Give the EE home a new location (dbhome_2 where as SE is dbhome_1)

Summary Page

6. Copy init file, password file and sqlnet.ora file (if used for COST) to appropriate location in the new EE home.

[oracle@rhel6m2 ~]$ cd /opt/app/oracle/product/11.2.0/dbhome_2/dbs
[oracle@rhel6m2 dbs]$  cp /opt/app/oracle/product/11.2.0/dbhome_1/dbs/orapwstd11g22 .
[oracle@rhel6m2 dbs]$  cp /opt/app/oracle/product/11.2.0/dbhome_1/dbs/initstd11g22.ora .

[oracle@rhel6m2 ~]$ cd /opt/app/oracle/product/11.2.0/dbhome_2/network/admin
[oracle@rhel6m2 dbs]$  cp /opt/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora .

Make sure this is done on all the nodes.




7. Patch the newly installed EE home (refer Case 2.1 section of the metalink note 1494646.1). Use GI_Home to run the opatch auto to avoid the following

Invoking utility "saveconfigurationsnapshot"
 UtilSession failed: This utility only be supported for GI home. Please specify the Oracle Home to the Grid Infrastructure (GI) Home "/opt/app/11.2.0/grid" by option -oh or set it in ORACLE_HOME environment variable

Following could be seen when patch is successfully applied.

/opt/app/11.2.0/grid/OPatch/opatch auto `pwd` -oh /opt/app/oracle/product/11.2.0/dbhome_2 -ocmrf ocm.rsp
 
 Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
patch /usr/local/patches/16619898/custom/server/16619898  apply successful for home  /opt/app/oracle/product/11.2.0/dbhome_2
patch /usr/local/patches/16619892  apply successful for home  /opt/app/oracle/product/11.2.0/dbhome_2

Compare the patch applied on EE home is same as SE home checked earlier on step 4

[oracle@rhel6m1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -local -oh /opt/app/oracle/product/11.2.0/dbhome_2 | grep Patch
Oracle Interim Patch Installer version 11.2.0.3.4
OPatch version    : 11.2.0.3.4
Patch  16619892     : applied on Thu Aug 15 10:34:09 BST 2013
Unique Patch ID:  16346737
Patch description:  "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch  16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  16619898     : applied on Thu Aug 15 10:27:19 BST 2013
Unique Patch ID:  16376391
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.7 (16742216)"
OPatch succeeded.

8. Verify the oracle executable in the EE home has the correct permissions. The oracle executable permission in the SE home were

oracle@rhel6m1 bin]$ ls -l oracle*
-rwsr-s--x. 1 oracle asmadmin 220193582 Jul 31 13:44 oracle
-rwsr-s--x. 1 oracle asmadmin 220113336 Apr 23 17:37 oracleO

while the newly installed EE had the wrong group ownership.

[oracle@rhel6m1 bin]$ ls -l oracle*
-rwsr-s--x. 1 oracle oinstall 232617534 Aug 15 10:35 oracle
-rwsr-s--x. 1 oracle oinstall 232399041 Aug 14 15:45 oracleO

Having this wrong group permission could result in following error when starting the database with the EE home (output from alert log)

SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database std11g2 and diskgroup resource ora.DATA.dg is established
Errors in file /opt/app/oracle/diag/rdbms/std11g2/std11g21/trace/std11g21_ora_24840.trc  (incident=67209):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/app/oracle/diag/rdbms/std11g2/std11g21/incident/incdir_67209/std11g21_ora_24840_i67209.trc
Use ADRCI or Support Workbench to package the incident.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 24840

To fix the issue set correct ownerships on the oracle executable.

[grid@rhel6m1 ~]$ $GI_HOME/bin/setasmgidwrap o=/opt/app/oracle/product/11.2.0/dbhome_2/bin/oracle
[grid@rhel6m1 ~]$ $GI_HOME/bin/setasmgidwrap o=/opt/app/oracle/product/11.2.0/dbhome_2/bin/oracleO

[oracle@rhel6m1 bin]$ ls -l oracle*
-rwsr-s--x. 1 oracle asmadmin 232617534 Aug 15 10:35 oracle
-rwsr-s--x. 1 oracle asmadmin 232399041 Aug 14 15:45 oracleO

More on kfioTranslateIO03 is available on the metalink note ORA-00600 [kfioTranslateIO03] [17090] (Doc ID 1336846.1)
9. Stop all but one instance.

[oracle@rhel6m1 ~]$ srvctl stop instance -d std11g2 -i std11g22

10. Set cluster_database=false to start the database in exclusive mode in the next start up.

SQL> alter system set cluster_database=FALSE scope=spfile sid='*';
System altered.

Once the change is done stop the remaining instance as well.
11. Change the oracle home associated with the database in the OCR.

[oracle@rhel6m1 ~]$ srvctl config database -d std11g2
Database unique name: std11g2
Database name: std11g2
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1 <--- SE Home

[oracle@rhel6m1 ~]$ srvctl modify database -d std11g2 -o /opt/app/oracle/product/11.2.0/dbhome_2

[oracle@rhel6m1 ~]$ srvctl config database -d std11g2
Database unique name: std11g2
Database name: std11g2
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_2  <--- EE Home

12. Change environment variables (eg. ORACLE_HOME, PATH values) to point to EE home and start the database in exclusive mode. Make sure sqlplus used is from EE home.

[oracle@rhel6m1 ~]$ which sqlplus
/opt/app/oracle/product/11.2.0/dbhome_2/bin/sqlplus

[oracle@rhel6m1 ~]$ sqlplus / as sysdba
SQL> startup
SQL> @?/rdbms/admin/catalog.sql
SQL> @?/rdbms/admin/catproc.sql
SQL> @?/rdbms/admin/utlrp

Monitor the alert log for any start up issues. Verify there are no object with errors or issues while compiling during the utrl execute.

SQL> select COUNT(DISTINCT(obj#)) "OBJECTS WITH ERRORS" from utl_recomp_errors;

OBJECTS WITH ERRORS
-------------------
                  0
1 row selected.

SQL> select COUNT(*) "ERRORS DURING RECOMPILATION" from utl_recomp_errors;

ERRORS DURING RECOMPILATION
---------------------------
                          0
1 row selected.

Also verify the registry component status

SQL> select comp_id,comp_name,version,status,schema from dba_registry;

COMP_ID    COMP_NAME                                VERSION                        STATUS     SCHEMA
---------- ---------------------------------------- ------------------------------ ---------- --------
XDB        Oracle XML Database                      11.2.0.3.0                     VALID      XDB
OWM        Oracle Workspace Manager                 11.2.0.3.0                     VALID      WMSYS
EXF        Oracle Expression Filter                 11.2.0.3.0                     VALID      EXFSYS
RUL        Oracle Rules Manager                     11.2.0.3.0                     VALID      EXFSYS
CATALOG    Oracle Database Catalog Views            11.2.0.3.0                     VALID      SYS
CATPROC    Oracle Database Packages and Types       11.2.0.3.0                     VALID      SYS
JAVAVM     JServer JAVA Virtual Machine             11.2.0.3.0                     VALID      SYS
XML        Oracle XDK                               11.2.0.3.0                     VALID      SYS
CATJAVA    Oracle Database Java Packages            11.2.0.3.0                     VALID      SYS
RAC        Oracle Real Application Clusters         11.2.0.3.0                     VALID      SYS

Fix any issues that exists.
13. Set cluster_database to true and shutdown the instance.

SQL> alter system set cluster_database=true scope=spfile sid='*'; 
SQL> shutdown immediate;

Start all the instance using srvctl. Make sure using the srvctl from EE home

[oracle@rhel6m1 ~]$ which srvctl
/opt/app/oracle/product/11.2.0/dbhome_2/bin/srvctl

[oracle@rhel6m1 ~]$ srvctl start database -d std11g2

Once the db is started /etc/oratab will be upated with the new oracle home

std11g2:/opt/app/oracle/product/11.2.0/dbhome_2:N               # line added by Agent

14. Remove standard edition specific settings or configuration that are no longer needed. For example drop any triggers used for enabling use of stored outlines or scheduling of statspack snapshots.
15. If diagnostic and tuning pack license are available make sure it's set

SQL> show parameter control
control_management_pack_access       string      DIAGNOSTIC+TUNING

Also verify that AWR snapshots are taken each hour.
16. Use an EE option to verify EE is being used by the database. Easiest way is to create a bitmap index which only available with EE.

SQL> create bitmap index aidx on x(a);
Index created.

If it is on SE following error message will be shown

SQL> create bitmap index aidx on x(a);
create bitmap index aidx on x(a)
*
ERROR at line 1:
ORA-00439: feature not enabled: Bit-mapped indexes

17. Finally remove the old SE homes either using deinstall tool or detaching from inventory and deleting

$ORACLE_HOME/oui/bin/runInstaller -detachHome ORACLE_HOME=SE HOME
rm -rf SE HOME

Useful metalink notes
How to Convert Database from Standard to Enterprise Edition ? [117048.1]
Converting An Enterprise Edition Database To Standard Edition [139642.1]
How to Convert a RAC database from Standard Edition (SE) to Enterprise Edition (EE)? [451981.1]

java.sql.SQLException: Could not commit with auto-commit set on When Using 12c JDBC Driver

When using the 12c JDBC driver calling commit on a connection that has auto commit true will result in following exception thrown on the client side

java.sql.SQLException: Could not commit with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:4439)
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:4486)

Even though the exception is thrown the SQL will get executed. Same however would execute without any error when jdbc driver 11.2 is used. Below java code demonstrate the test case.

public static void main(String[] args) {
        try {
            OracleDataSource ds = new OracleDataSource();
            ds.setUser("asanga");
            ds.setPassword("asa");
            ds.setURL("jdbc:oracle:thin:@192.168.0.93:9120:std11g21");

            Connection con = ds.getConnection();

//            con.setAutoCommit(false);
            System.out.println("Auto commit status : " + con.getAutoCommit());
            DatabaseMetaData meta = con.getMetaData();

            System.out.println("Driver Name " + meta.getDriverName());
            System.out.println("Driver Version " + meta.getDriverVersion());
            System.out.println("Driver Major Version " + meta.getDriverMajorVersion());
            System.out.println("Driver Minor Version " + meta.getDriverMinorVersion());
            System.out.println("Database Major Version " + meta.getDatabaseMajorVersion());
            System.out.println("Database Minor Version " + meta.getDatabaseMinorVersion());
            PreparedStatement pr = con.prepareStatement("insert into x values(?)");
            pr.setInt(1, 10);
            pr.execute();
            pr.close();
            con.commit();

        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }

Running with 12c Driver the the following output is seen on the client side and querying the table X will show a new row has been inserted successfully.

Auto commit status : true
Driver Name Oracle JDBC driver
Driver Version 12.1.0.1.0
Driver Major Version 12
Driver Minor Version 1
Database Major Version 12
Database Minor Version 1
java.sql.SQLException: Could not commit with auto-commit set on
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:4439)
        at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:4486)

However running with 11.2 driver doesn't give any exception

Auto commit status : true
Driver Name Oracle JDBC driver
Driver Version 11.2.0.3.0
Driver Major Version 11
Driver Minor Version 2
Database Major Version 12
Database Minor Version 1

One way to overcome the issue is to set auto commit false once the connection is checked out of the pool or not calling the commit. However for application where source code is not accessible upgrading to 12c driver could be problematic. According to Oracle this referrers to bug 15891607 which is not a bug but expected behavior. Sure enough the APIs also says the same that calling commit on a connection object with auto commit set true will cause an exception JDK 7 API and JDK 6 API. However this wasn't observed when the code is run so moving to 12c driver from 11g may require code changes in certain cases.

Related Posts
getBoolean in 12c JDBC Driver and java.sql.SQLException: Fail to convert to internal representation
Change in 12c JDBC Behavior - setDate & getDate Does Not Truncate Timestamp




Update on 2020-04-02

As per MOS 1564509.1 adding

-Doracle.jdbc.autoCommitSpecCompliant=false

system property will revert to the legacy behaviour. This could be useful when code changes are not possible.