Installing 12cR1 RAC without the use of the new features such as the
Oracle Flex ASM is pretty much same as
installing 11gR2 RAC with role separation. This post is not a step by step guide to installing 12cR1 RAC but list the main differences in the installation process compared to previous versions and pitfalls to look out for.
Among the noticeable changes and new features 12c doesn't support storing clusterware files in block devices as such provisions must be made either to store in the shared file systems or in ASM. Even if upgrading ocr files must be moved to ASM before the upgrade.
12c supports IPv6 for public IP and VIP. However this post uses IPv4 for public and VIP.
12c has also introduced several new administrative privileges for job role separation for database related tasks in data guard, key management and backup.
1. The RHEL 6 version used for the setup is RHEL 6 - U4
# uname -r
2.6.32-358.el6.x86_64
2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba
useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid
grid user was made part of the dba group as per 1084186.1. (If this issue is still there on 12c was not tested).
3.
Create user equivalence manually and do not wait for OUI to do this during the installation even though Oracle documentation says "
You can configure SSH from the OUI interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.". Without this manual user equivalence none of the cluvfy commands will complete successfully.
4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission. For GI_HOME
mkdir -p /opt/app/12.1.0/grid
chown -R grid:oinstall /opt/app
chown grid:oinstall /opt
chmod -R 775 /opt/app
chmod 775 /opt
For ORACLE_BASE
mkdir -p /opt/app/oracle
chown oracle:oinstall /opt/app/oracle
chmod 775 /opt/app/oracle
After running the root.sh the ownership of these directories (for GI_HOME) will be changed to root. But before the installation it must be owned by grid with oinstall as group and both user and group must have read,write and execute permissions.
5. Verify pre-req package list is installed. Follow
GI installation guide and metalink note
Requirements for Installing Oracle Database 12.1 on RHEL6 or OL6 64-bit (x86-64) [ID 1529864.1] for list of rpms required. Starting with 11.2.0.2 and including 12c "
the 32-bit packages are required only for 32-bit client installs". In this setup servers used had both 32-bit and 64-bit version were installed.
6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file
Host *
ForwardX11 no
8. Disable tty on .bashrc for both grid and oracle user
if [ -t 0 ]; then
stty intr ^C
fi
and also set umask value in .bashrc for grid and oracle user
umask 022
9. ASMLib is not used for this RAC setup. Instead block devices are used as ASM disk. Setup udev rules for block devices with following ownerships and permissions.
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
If scsi_id command is used to find the UUID then the command listed on the GI install documentation fails on RHEL 6.
# /sbin/scsi_id -g -s /block/sdb/sdb1
/sbin/scsi_id: invalid option -- 's'
-s is not a supported option on RHEL 6. Instead use the following
/sbin/scsi_id -g -u -d /dev/sdb
Further more Oracle documentation mentions with
external redundancy a minimum of 1 disk of 700MB size is sufficient to store both OCR and vote disk when not configuring cluster management repository. However it must be stated 700MB is not sufficient the disk space should be at least 1069MB.
In practice it is best to use normal or high redundancy on the disk group used to store OCR and vote disk.
10. Running cluvfy fails with the following message
ERROR:
Reference data is not available for verifying prerequisites on this operating system distribution
Verification cannot proceed
As mentioned in
upgrade to 12c post install redhat-release-6Server-1.noarch.rpm found in the pack available with
1514012.1. (Also refer
1567127.1). After the aforementioned rpm is installed cluvfy can continue without any issue. If the cluvfy or OUI cannot find the UUID for a disk during the disk share check following message will be shown. From cluvfy
Checking Devices for ASM...
ERROR: /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
rhel12c2,rhel12c1
Checking for shared devices...
PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodesFrom OUI
Error is due to test system using
VirtualBox to setup the RAC and partitions not returning an UUID. Installation could be continued ignoring this error. In a proper system where UUID is available the cluvfy would have the following messages when these check succeed.
Checking Devices for ASM...
Checking for shared devices...
Device Device Type
------------------------------------ ------------------------
/dev/sdb1 Disk
Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/sdb1" PASSED
Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/sdb1" PASSED
Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/sdb1" PASSED
Checking consistency of device size across all nodes...
Consistency check of device size for "/dev/sdb1" PASSED
Below is the output from running pre crsinst.
[grid@rhel12c1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel12c1,rhel12c2 -asm -asmgrp asmadmin -presence local -asmdev /dev/sdb1 -crshome /opt/app/12.1.0/grid -fixup -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "rhel12c1"
Destination Node Reachable?
------------------------------------ ------------------------
rhel12c1 yes
rhel12c2 yes
Result: Node reachability check passed from node "rhel12c1"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
rhel12c2 passed
rhel12c1 passed
Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
rhel12c1 passed
rhel12c2 passed
Verification of the hosts config file successful
Interface information for node "rhel12c1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.0.93 192.168.0.0 0.0.0.0 192.168.0.100 08:00:27:CB:D8:AE 1500
eth1 192.168.1.87 192.168.1.0 0.0.0.0 192.168.0.100 08:00:27:92:0B:69 1500
Interface information for node "rhel12c2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.0.94 192.168.0.0 0.0.0.0 192.168.0.100 08:00:27:9C:66:81 1500
eth1 192.168.1.88 192.168.1.0 0.0.0.0 192.168.0.100 08:00:27:1E:0E:A8 1500
Check: Node connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rhel12c1[192.168.0.93] rhel12c2[192.168.0.94] yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c1,rhel12c2
Check: TCP connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rhel12c1:192.168.0.93 rhel12c2:192.168.0.94 passed
Result: TCP connectivity check passed for subnet "192.168.0.0"
Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rhel12c1[192.168.1.87] rhel12c2[192.168.1.88] yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c1,rhel12c2
Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rhel12c1:192.168.1.87 rhel12c2:192.168.1.88 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"
Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel12c1 eth0:192.168.0.93
rhel12c2 eth0:192.168.0.94
Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel12c1 eth1:192.168.1.87
rhel12c2 eth1:192.168.1.88
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Node Name Status
------------------------------------ ------------------------
rhel12c1 passed
rhel12c2 passed
Result: Check for ASMLib configuration passed.
Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 4.4199GB (4634568.0KB) 4GB (4194304.0KB) passed
rhel12c1 4.4199GB (4634568.0KB) 4GB (4194304.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 4.2905GB (4498884.0KB) 50MB (51200.0KB) passed
rhel12c1 4.1755GB (4378320.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 5GB (5242872.0KB) 4.4199GB (4634568.0KB) passed
rhel12c1 5GB (5242872.0KB) 4.4199GB (4634568.0KB) passed
Result: Swap space check passed
Check: Free disk space for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr rhel12c2 / 21.3525GB 7.9635GB passed
/var rhel12c2 / 21.3525GB 7.9635GB passed
/etc rhel12c2 / 21.3525GB 7.9635GB passed
/opt/app/12.1.0/grid rhel12c2 / 21.3525GB 7.9635GB passed
/sbin rhel12c2 / 21.3525GB 7.9635GB passed
/tmp rhel12c2 / 21.3525GB 7.9635GB passed
Result: Free disk space check passed for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"
Check: Free disk space for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr rhel12c1 / 18.3523GB 7.9635GB passed
/var rhel12c1 / 18.3523GB 7.9635GB passed
/etc rhel12c1 / 18.3523GB 7.9635GB passed
/opt/app/12.1.0/grid rhel12c1 / 18.3523GB 7.9635GB passed
/sbin rhel12c1 / 18.3523GB 7.9635GB passed
/tmp rhel12c1 / 18.3523GB 7.9635GB passed
Result: Free disk space check passed for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"
Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
rhel12c2 passed exists(501)
rhel12c1 passed exists(501)
Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "grid"
Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
rhel12c2 passed exists
rhel12c1 passed exists
Result: Group existence check passed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
rhel12c2 passed exists
rhel12c1 passed exists
Result: Group existence check passed for "dba"
Check: Group existence for "asmadmin"
Node Name Status Comment
------------ ------------------------ ------------------------
rhel12c2 passed exists
rhel12c1 passed exists
Result: Group existence check passed for "asmadmin"
Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Status
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c2 yes yes yes yes passed
rhel12c1 yes yes yes yes passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed
Check: Membership of user "grid" in group "dba"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 yes yes yes passed
rhel12c1 yes yes yes passed
Result: Membership check for user "grid" in group "dba" passed
Check: Membership of user "grid" in group "asmadmin"
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 yes yes yes passed
rhel12c1 yes yes yes passed
Result: Membership check for user "grid" in group "asmadmin" passed
Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 3 3,5 passed
rhel12c1 3 3,5 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 hard 65536 65536 passed
rhel12c1 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 soft 1024 1024 passed
rhel12c1 soft 1024 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 hard 16384 16384 passed
rhel12c1 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 soft 2047 2047 passed
rhel12c1 soft 2047 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 x86_64 x86_64 passed
rhel12c1 x86_64 x86_64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 2.6.32-358.el6.x86_64 2.6.32 passed
rhel12c1 2.6.32-358.el6.x86_64 2.6.32 passed
Result: Kernel version check passed
Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 3010 3010 250 passed
rhel12c2 3010 3010 250 passed
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 385280 385280 32000 passed
rhel12c2 385280 385280 32000 passed
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 3010 3010 100 passed
rhel12c2 3010 3010 100 passed
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 128 128 128 passed
rhel12c2 128 128 128 passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 68719476736 68719476736 2372898816 passed
rhel12c2 68719476736 68719476736 2372898816 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 4096 4096 4096 passed
rhel12c2 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 4294967296 4294967296 463456 passed
rhel12c2 4294967296 4294967296 463456 passed
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 6815744 6815744 6815744 passed
rhel12c2 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
rhel12c2 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 4194304 4194304 262144 passed
rhel12c2 4194304 4194304 262144 passed
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 4194304 4194304 4194304 passed
rhel12c2 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 1048576 1048576 262144 passed
rhel12c2 1048576 1048576 262144 passed
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 2097152 2097152 1048576 passed
rhel12c2 2097152 2097152 1048576 passed
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rhel12c1 3145728 3145728 1048576 passed
rhel12c2 3145728 3145728 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed
rhel12c1 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed
Result: Package existence check passed for "binutils"
Check: Package existence for "compat-libcap1"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
rhel12c1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
Result: Package existence check passed for "compat-libcap1"
Check: Package existence for "compat-libstdc++-33(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
rhel12c1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "libgcc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 libgcc(x86_64)-4.4.7-3.el6 libgcc(x86_64)-4.4.4 passed
rhel12c1 libgcc(x86_64)-4.4.7-3.el6 libgcc(x86_64)-4.4.4 passed
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 libstdc++(x86_64)-4.4.7-3.el6 libstdc++(x86_64)-4.4.4 passed
rhel12c1 libstdc++(x86_64)-4.4.7-3.el6 libstdc++(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 libstdc++-devel(x86_64)-4.4.7-3.el6 libstdc++-devel(x86_64)-4.4.4 passed
rhel12c1 libstdc++-devel(x86_64)-4.4.7-3.el6 libstdc++-devel(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
rhel12c1 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
Result: Package existence check passed for "sysstat"
Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 gcc-4.4.7-3.el6 gcc-4.4.4 passed
rhel12c1 gcc-4.4.7-3.el6 gcc-4.4.4 passed
Result: Package existence check passed for "gcc"
Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.4 passed
rhel12c1 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.4 passed
Result: Package existence check passed for "gcc-c++"
Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 ksh-20100621-19.el6 ksh-... passed
rhel12c1 ksh-20100621-19.el6 ksh-... passed
Result: Package existence check passed for "ksh"
Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 make-3.81-20.el6 make-3.81 passed
rhel12c1 make-3.81-20.el6 make-3.81 passed
Result: Package existence check passed for "make"
Check: Package existence for "glibc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 glibc(x86_64)-2.12-1.107.el6 glibc(x86_64)-2.12 passed
rhel12c1 glibc(x86_64)-2.12-1.107.el6 glibc(x86_64)-2.12 passed
Result: Package existence check passed for "glibc(x86_64)"
Check: Package existence for "glibc-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 glibc-devel(x86_64)-2.12-1.107.el6 glibc-devel(x86_64)-2.12 passed
rhel12c1 glibc-devel(x86_64)-2.12-1.107.el6 glibc-devel(x86_64)-2.12 passed
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "libaio(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
rhel12c1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "libaio-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
rhel12c1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio-devel(x86_64)"
Check: Package existence for "nfs-utils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 nfs-utils-1.2.3-36.el6 nfs-utils-1.2.3-15 passed
rhel12c1 nfs-utils-1.2.3-36.el6 nfs-utils-1.2.3-15 passed
Result: Package existence check passed for "nfs-utils"
Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
Node Name Port Number Protocol Available Status
---------------- ------------ ------------ ------------ ----------------
rhel12c2 6200 TCP yes successful
rhel12c1 6200 TCP yes successful
rhel12c2 6100 TCP yes successful
rhel12c1 6100 TCP yes successful
Result: Port availability check passed for ports "6200,6100"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
rhel12c2 passed
rhel12c1 passed
Check for consistency of root user's primary group passed
Check: Package existence for "cvuqdisk"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rhel12c2 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
rhel12c1 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
Result: Package existence check passed for "cvuqdisk"
Checking Devices for ASM...
ERROR: /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
rhel12c2,rhel12c1
Checking for shared devices...
PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes
UDev attributes check for ASM Disks started...
Checking udev settings for device "/dev/sdb1"
Device Owner Group Permissions Result
---------------- ------------ ------------ ------------ ----------------
sdb1 grid asmadmin 0660 passed
sdb1 grid asmadmin 0660 passed
Result: UDev attributes check passed for ASM Disks
Result: Devices check for ASM passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
rhel12c2 passed does not exist
rhel12c1 passed does not exist
Result: User "grid" is not part of "root" group. Check passed
Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rhel12c2 0022 0022 passed
rhel12c1 0022 0022 passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
Node Name Status
------------------------------------ ------------------------
rhel12c1 passed
rhel12c2 passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
Check for integrity of file "/etc/resolv.conf" passed
Check: Time zone consistency
Result: Time zone consistency check passed
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
Checking daemon "avahi-daemon" is not configured and running
Check: Daemon "avahi-daemon" not configured
Node Name Configured Status
------------ ------------------------ ------------------------
rhel12c2 no passed
rhel12c1 no passed
Daemon not configured check passed for process "avahi-daemon"
Check: Daemon "avahi-daemon" not running
Node Name Running? Status
------------ ------------------------ ------------------------
rhel12c2 no passed
rhel12c1 no passed
Daemon not running check passed for process "avahi-daemon"
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passed
NOTE:
No fixable verification failures to fix
Pre-check for cluster services setup was successful.Despite the issues on the ASM disk share cluvfy reports pre-reqs to be successful.
11. Start the clusterware installation by executing runInstaller.
12c provides many "use for" options for NICs. Since this is not setting up Flex ASM for eth1 select private. Oracle documentation states "
Oracle Flex ASM can use either the same private networks as Oracle Clusterware, or use its own dedicated private networks. Each network can be classified PUBLIC, ASM & PRIVATE, PRIVATE, or ASM."
Selecting no prompts the below shown message. If GI management repository is installed then ASM space requirement would be higher. Refer Oracle documentation and
12c upgrade post for space requirements for management repository
12. When prompted execute the root.sh scripts. Below is the output from running the root.sh on first node.
[root@rhel12c1 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /opt/app/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:35:28 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2013/08/20 17:36:29 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
ASM created and started successfully.
Disk Group clusterdg created successfully.
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c09b02f830b64f49bf06b75f49b272ee.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE c09b02f830b64f49bf06b75f49b272ee (/dev/sdb1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c1'
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:44:10 CLSRSC-343: Successfully started Oracle clusterware stack
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
2013/08/20 17:45:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeededRunning root.sh on second node.
[root@rhel12c2 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /opt/app/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:45:54 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2013/08/20 17:46:24 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c2'
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:52:08 CLSRSC-343: Successfully started Oracle clusterware stack
2013/08/20 17:52:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded13. Once root.sh is completed click OK on the execute script dialog to continue with the remaining configurations after which clusterware installation is complete.
14. Verify the cluster installation with cluvfy post crsinst
cluvfy stage -post crsinst -n rhel12c1,rhel12c2 -verbose
15. Use ASMCA to create additional ASM disk groups that will be used for data and flash recovery when database is setup. Cluvfy could be used to verify if the disk are shared
cluvfy comp ssa -n rhel12c1,rhel12c2 -s /dev/sdc1,/dev/sdd1 -t data -r 12.1 -verbose
Next step is to
install the Oracle Database software.
Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)
