Oracle Invoked Out-of-Memory Killer (oom-killer)

One node on a 2-node RAC (11.1.0.7) crashed and following could seen on the /var/log/messages

Feb 11 09:46:01 server02 logger: Oracle CSSD waiting for OPROCD to start
Feb 11 22:46:21 server02 kernel: oracle invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Feb 11 22:46:21 server02 kernel:
Feb 11 22:46:21 server02 kernel: Call Trace:
Feb 11 22:46:21 server02 kernel:  [] out_of_memory+0x8e/0x2f3
Feb 11 22:46:21 server02 kernel:  [] __alloc_pages+0x27f/0x308
Feb 11 22:46:21 server02 kernel:  [] getnstimeofday+0x10/0x28
Feb 11 22:46:21 server02 kernel:  [] __do_page_cache_readahead+0x96/0x179
Feb 11 22:46:21 server02 kernel:  [] filemap_nopage+0x14c/0x360
Feb 11 22:46:21 server02 kernel:  [] __handle_mm_fault+0x1fa/0xfaa
Feb 11 22:46:21 server02 kernel:  [] do_page_fault+0x4cb/0x874
Feb 11 22:46:21 server02 kernel:  [] thread_return+0x62/0xfe
Feb 11 22:46:21 server02 kernel:  [] error_exit+0x0/0x84
Feb 11 22:46:21 server02 kernel:
Feb 11 22:46:23 server02 kernel: Mem-info:
Feb 11 22:46:23 server02 kernel: Node 0 DMA per-cpu:
Feb 11 22:46:23 server02 kernel: cpu 0 hot: high 0, batch 1 used:0
Feb 11 22:46:24 server02 kernel: cpu 0 cold: high 0, batch 1 used:0
Feb 11 22:46:25 server02 kernel: cpu 1 hot: high 0, batch 1 used:0
Feb 11 22:46:26 server02 kernel: cpu 1 cold: high 0, batch 1 used:0
Feb 11 22:46:26 server02 kernel: cpu 2 hot: high 0, batch 1 used:0
Feb 11 22:46:26 server02 kernel: cpu 2 cold: high 0, batch 1 used:0
Feb 11 22:46:27 server02 kernel: cpu 3 hot: high 0, batch 1 used:0
Feb 11 22:46:27 server02 kernel: cpu 3 cold: high 0, batch 1 used:0
Feb 11 22:46:28 server02 kernel: Node 0 DMA32 per-cpu:
Feb 11 22:46:29 server02 kernel: cpu 0 hot: high 186, batch 31 used:16
Feb 11 22:46:29 server02 kernel: cpu 0 cold: high 62, batch 15 used:29
Feb 11 22:46:29 server02 kernel: cpu 1 hot: high 186, batch 31 used:26
Feb 11 22:46:29 server02 kernel: cpu 1 cold: high 62, batch 15 used:14
Feb 11 22:46:29 server02 kernel: cpu 2 hot: high 186, batch 31 used:27
Feb 11 22:46:29 server02 kernel: cpu 2 cold: high 62, batch 15 used:32
Feb 11 22:46:29 server02 kernel: cpu 3 hot: high 186, batch 31 used:9
Feb 11 22:46:30 server02 kernel: cpu 3 cold: high 62, batch 15 used:59
Feb 11 22:46:30 server02 kernel: Node 0 Normal per-cpu:
Feb 11 22:46:30 server02 kernel: cpu 0 hot: high 186, batch 31 used:80
Feb 11 22:46:30 server02 kernel: cpu 0 cold: high 62, batch 15 used:54
Feb 11 22:46:30 server02 kernel: cpu 1 hot: high 186, batch 31 used:38
Feb 11 22:46:30 server02 kernel: cpu 1 cold: high 62, batch 15 used:55
Feb 11 22:46:30 server02 kernel: cpu 2 hot: high 186, batch 31 used:30
Feb 11 22:46:30 server02 kernel: cpu 2 cold: high 62, batch 15 used:58
Feb 11 22:46:31 server02 kernel: cpu 3 hot: high 186, batch 31 used:38
Feb 11 22:46:31 server02 kernel: cpu 3 cold: high 62, batch 15 used:51
Feb 11 22:46:31 server02 kernel: Node 0 HighMem per-cpu: empty
Feb 11 22:46:31 server02 kernel: Free pages:       76148kB (0kB HighMem)

Oracle provides a metalink note for lowMem region memory pressure (452326.1) but this crash is due to highMem region so note wasn't much help. If it's not related to a bug (551991.1) then the root cause could be due to memory pressure(1502301.1). It must be verified that system is indeed under memory pressure. System statistics (either with sysstat or OSWatcher) could be used to check if memory consumption differed from normal due to any recent application changes.
Other solution includes setting vm.lower_zone_protection (only on 32-bit systems) or vm.min_free_kbytes kernel parameter depending on the architecture.

Useful Metalink notes
Linux: Out-of-Memory (OOM) Killer [452000.1]
How to Check Whether a System is Under Memory Pressure [1502301.1]
Linux Kernel: The SLAB Allocator [434351.1]
BUG 6167888: RMAN-10038 ERROR ON LINUX AFTER CREATING A LARGE BACKUPSET ON NFS [551991.1]
Linux Kernel Lowmem Pressure Issues and Kernel Structures [452326.1]

Finding Sector Size of a Disk

Sector size of a disk could be found out with the blockdev command in linux

blockdev
Usage:
  blockdev -V
  blockdev --report [devices]
  blockdev [-v|-q] commands devices
Available commands:
        --getsz (get size in 512-byte sectors)
        --setro (set read-only)
        --setrw (set read-write)
        --getro (get read-only)
        --getss (get sectorsize)
        --getbsz        (get blocksize)
        --setbsz BLOCKSIZE      (set blocksize)
        --getsize       (get 32-bit sector count)
        --getsize64     (get size in bytes)
        --setra READAHEAD       (set readahead)
        --getra (get readahead)
        --flushbufs     (flush buffers)
        --rereadpt      (reread partition table)
        --rmpart PARTNO (disable partition)
        --rmparts       (disable all partitions)

To get the sector size use getss option with the device

/sbin/blockdev --getss /dev/sdd1
512

Many options could be combined together as well such as finding out block size as well as the sector size

/sbin/blockdev --getss --getbsz /dev/sdd1
512
2048

Knowing the sector size may be useful when dealing with new features of ASM.

In windows sector size could be found out with

wmic DISKDRIVE get bytespersector, caption
BytesPerSector  Caption
512             TOSHIBA MK1216GSY

Installing 11.2.0.3 on RHEL 6

Oracle has recently certified RHEL6 for 11.2.0.3 or higher version. Read
Certification Information for Oracle Database on Linux x86-64 [ID 1304727.1] for more information.

The requirements for RHEL6 are given in the following metalink note
Requirements for Installing Oracle 11gR2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1]

Oracle documentation has also been updated to reflect RHEL6 requirements. One minor adjustment is needed to run the 11.2.0.3 installer if not following pre-req failure will appear. 

To remove this change the CV_ASSUME_DISTID entry in cvu_config file (located in database/stage/cvu/cv/admin) from
CV_ASSUME_DISTID=OEL4
to
CV_ASSUME_DISTID=OEL6

More information regarding this is available in
Installing 11.2.0.3 64-bit (x86-64) on RHEL6 Reports That Packages "elfutils-libelf-devel-0.97" and "pdksh-5.2.14" are missing (PRVF-7532) [ID 1454982.1]

Apart from this there is no other considerable difference when it comes to installing Oracle in RHEL 6 compared to other RHEL versions.

When it comes to asmlibs they will have to be downloaded from unbreakable linux site and will not be issued for kernels distributed by RedHat. More on this
Oracle ASMLib Software Update Policy for Red Hat Enterprise Linux Supported by Red Hat [ID 1089399.1]

Same goes for ocfs2 and REHL 6 as well.
Oracle Cluster File System (OCFS2) Software Support and Update Policy for Red Hat Enterprise Linux Supported by Red Hat [ID 1253272.1]

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11.2.0.4 Standalone Server with ASM and Role Separation on RHEL 7

Linux Root Exploit CVE-2012-0056

A Linux vulnerability has been identified which allows a user to gain root user privilege. This is caused by "failure of the Linux kernel to properly restrict access to the "/proc//mem" file".

Computer world has a detail article about this (Same article available on CIO).

Redhat has a support note regarding this vulnerability and a test code to check if the system is vulnerable.

Resizing VirtualBox and Linux Partitions with GParted

VirtualBox 4 has an option to modify the logical size of the virtual disk after it's been created without affecting the physical size. The size specified in this option is the new overall size of the disk not the increment. If the current size is 100MB and resizing to 200MB would make the disk 200MB not 300MB (100 + 200). Once the size is increased it cannot be decreased (yet).

Current size of the disk

-rw------- 1 root root  18G Jan 27 16:43 rac4.vdi

The size of disk seen by the OS using this virutal disk

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Size of the partitions

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              16G   13G  1.8G  88% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                1006M     0 1006M   0% /dev/shm

Increase the size with modifyhd option

VBoxManage modifyhd rac4.vdi --resize 40960
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Physical size is unaffected. This will grow as the space is used.

-rw------- 1 root root  18G Jan 27 16:45 rac4.vdi

But the size of the disk seen by the OS has changed

Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

The additional space is unallocated and linux partitions are not affected after this increase.

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              16G   13G  1.8G  88% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                1006M     0 1006M   0% /dev/shm

Using the Gnome Partition Editor (GParted) it is possible to resize the linux partition and allocate the unused space to it effectively increasing the size of it.

Download the Live CD iso of gparted and boot the virtual box server from it.

Select the default option and complete the startup.

GParted shows the size of current partitions (15.9 GB) and the unallocated space (20 GB).

Right click on the partition to resize and select resize/move item from the popup menu.

Resize the partition by either dragging the highlighted box or adjusting the numerical spinners.

Click apply to complete the resizing.

Restart the virtual server and see the size of linux partition changed.

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              35G   13G   21G  39% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                1006M     0 1006M   0% /dev/shm

Size of the new partition is 35 GB (15 GB previously allocated and 20 GB added with GParted).

Configuring NTP Service on Linux

Servers' time fall out of sync with one another after some time. This is problematic if the time gap between each server is wide such that it affects the execution of business logic. In RAC configuration this could even lead to node eviction.

The solution is to sync all servers with a NTP server that keep accurate time. For this to work the servers must be able to connect to the internet to access these NTP servers. But for security reason this may not always be possible. However it is possible to have a setup as shown below where only one server connect to the NTP servers and all others get their time sync from this "internal" server without having to connect to the internet.

It is assumed ntp is installed on the Linux servers if not install.

1. Backup the current /etc/ntp.conf file on all servers.

mv /etc/ntp.conf /etc/ntp.conf.orig

2. On the server that is going to act as the internal NTP server create a new ntp.conf file (make sure it's owned by root and permissions are 644) in /etc and add the following entries. Important parts are shown in bold

server  ntpserver1.org           
server  ntpserver2.org           

restrict ntpserver1.org  mask 255.255.255.255 nomodify notrap noquery
restrict ntpserver1.org  mask 255.255.255.255 nomodify notrap noquery

restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap

restrict 127.0.0.1
restrict -6 ::1

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

driftfile /var/lib/ntp/drift

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

ntpserver1/2.org are the NTP servers that are being used for synchronization. (netserver1/2.org used here is just an example).

The restirct line tells what is allowed for the NTP servers, in this case NTP servers are not allowed to modify nor query the local server. The mask 255.255.255.255 limits access to the single IP address of the remote NTP servers.

Third line is the key for allowing other servers to get the time from this server. This is done by removing the noquery line from the restrict and specifiying computers coming from which network segment are allowed to query. In this case all the servers in the local network would be able to get their time sync from this server.

3. Leave the other options as it is.

4. If the server time is lagging far behind the standard time, then use the ntpdate command to get the time updated allowing ntp service to start with a small time lag. To do this first stop the ntp service

/etc/init.d/ntpd stop

and then run

ntpdate -u ntpserver1.org
16 Nov 12:58:49 ntpdate[13104]: adjust time server 129.67.1.160 offset -0.070418 sec

Run the above command 2-3 times.

5. Start the ntp service and monitor the ntp synchronization in progress

/etc/init.d/ntpd start

watch -n 5 ntpq -p

Every 5.0s: ntpq -p             Wed Nov 16 13:08:44 2011

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntpserver1.org 192.6.2.82     2 u    7   64  377    3.732  -67.808   0.520
+ntpserver2.org 192.6.2.74     2 u   61   64  377    3.755  -67.672  17.760
 LOCAL(0)        .LOCL.          10 l   53   64  377    0.000    0.000   0.001

6. On other servers (servers get their time sync from the internal NTP server) eg. App and DB servers on the above diagram, create a new /etc/ntp.conf file and add the following entries

server  internal_ntp.server.net

restrict internal_ntp.server.net   mask 255.255.255.255 nomodify notrap noquery

restrict 127.0.0.1
restrict -6 ::1

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

driftfile /var/lib/ntp/drift

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

In this configuraiton the ntp server would be the hostname or the IP of the internal ntp server.

7. If this is a RAC server then before start the ntp service add -x to /etc/sysconfig/ntpd file(metalink note 551704.1)

# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

8. Start the ntp service and monitor the synchronization progress

Every 5.0s: ntpq -p Wed Nov 16 13:19:40 2011

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 internal_ntp_IP    163.1.2.160      3 u    1   64    1    0.279    0.074   0.001
 LOCAL(0)        .LOCL.          10 l    -   64    0    0.000    0.000   0.001

Useful metalink notes

NTP Information and Setup [ID 1010136.1]
Ntpd Does not Use Defined NTP Server [ID 1178614.1]
An Example NTP Client Configuration to use with Oracle Clusterware 11gR2 [ID 1104473.1]
Linux OS Service ntpd [ID 551704.1]
How to Set Up a Network Time Protocol (NTP) Client in Solaris [ID 1005887.1]
CTSSD Runs in Observer Mode Even Though No Time Sync Software is Running [ID 1054006.1]
NTP leap second event causing Oracle Clusterware node reboot [ID 759143.1]

How to mount an ISO image file

1. Create a directory (mount point) fro the iso file.

2. As root mount the file, in this case default /media directory has been used as the mount point

mount -o loop rhel-server-5.5-x86_64-dvd.iso /media

3. To access the files inside the iso cd to /media.

A loop device is a pseudo-device that makes a file accessible as a block device

Warning of HWaddr when network restarted with bonding

Following warning could be observed on /var/log/messages when the network is restarted

Nov  4 08:18:09 db1 kernel: bonding: bond0: enslaving eth1 as an active interface with an up link.
Nov  4 08:18:12 db1 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
Nov  4 08:18:12 db1 kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex
Nov  4 08:22:55 db1 kernel: bonding: bond0: Removing slave eth0
Nov  4 08:22:55 db1 kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - F0:4D:A2:06:BC:7B - is still in use by bond0.
Set the HWaddr of eth0 to a different address to avoid conflicts.
Nov  4 08:22:55 db1 kernel: bonding: bond0: releasing active interface eth0
Nov  4 08:22:55 db1 kernel: bonding: bond0: Removing slave eth1
Nov  4 08:22:55 db1 kernel: bonding: bond0: releasing active interface eth1
Nov  4 08:22:55 db1 kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready
Nov  4 08:22:55 db1 kernel: bonding: bond0: Adding slave eth0.

According to RedHat support document DOC-37419 this is a warning only doesn't indicate an issue.

It could be avoided by adding a MACADDR to the bonding interface. If the bonding interface is bond0 then adding following line to ifcfg-bond0 will make the warning disappear.

MACADDR=AA:BB:CC:11:22:33

The MAC address must be unique to that network segment.

Root cause for this is, when a MAC address is not defined bonding interface will use one of the slaves MAC address, and when that slave interface goes down bonding interface still prefers to use the same MAC address regardless other slave is now the active slave.

Public YUM Server

Oracle has a public yum server which offers a free and convenient way to install packages from the Enterprise Linux and Oracle VM installation media via a yum client.

Oracle Enterprise Linux 4, Update 6 or Newer

# cd /etc/yum.repos.d
# mv Oracle-Base.repo Oracle-Base.repo.disabled
# wget http://public-yum.oracle.com/public-yum-el4.repo

Oracle Enterprise Linux 5

# cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-el5.repo

Oracle Enterprise Linux 6

# cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-ol6.repo

Oracle VM 2

# cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-ovm2.repo

Enable the appropriate repository by editing the yum configuration file

* Open the yum configuration file in a text editor
* Locate the section in the file for the repository you plan to update from, e.g. [el4_u6_base]
* Change enabled=0 to enabled=1

Test with

yum list

Oracle10gR2 On RHEL 5/OEL 5 (x86_64)

Changes to the packages and pre-req parameters

more on metalink 421308.1

Jan 19, 2009:

1. The required packages this article have been changed from Release Notes because of non-existent names and/or versions given in Release Notes.
a) Packages versions changed
1. compat-db-4.1 to compat-db-4.2
2. glibc-2.3 (i386) to glibc-2.5 (i386)
3. libstdc++-3.4.3-9.EL4 to libstdc++-4.1.1
4. libstdc++-devel-3.4.3-22.1 to libstdc++-devel-4.1.1

b) Package names changed
1. pdksh-5.2 to ksh-20060214-1.4
2. gnome-libs to libgnome-2.16.0 and libgnomeui-2.16.0

c) Packages Added
1. libXp-1.0.0 (i386) due to Bug 7680459

2. Updated kernel parameter net.ipv4.ip_local_port_range as per correction request given by Oracle development team.

Feb 11, 2009:

1. Updated the physical memory required as 1GB

2. Removed swap space requirements when physical memory < 1GB

3. Added information about disabling SELinux
Feb 27, 2009:

1. Removed kernel parameter fs.file-max = 65536 , because the default value 202804 is more than 65536.

2. Corrected typo:
glibc-2.5-12 (i386) to glibc-2.5-12 (i686)

Mar 2, 2009:
Removed kernel parameters net.ipv4.tcp_wmem and net.ipv4.tcp_rmem as they are not required. (Reference: unpublished Bugs 7647281 and 7647303)

Script for creating bonded interfaces

Usage:

bond.sh interface1 interface2 bondname IP boradcastIP netmask network mode link_monitoring_frequency(miimon)

./bond.sh eth5 eth6 bond0 192.168.0.100 192.168.0.255 255.255.255.0 192.168.0.0 6 100

Copy the text below to create the bond.sh script

#! /bin/sh

if [ $# -ne 9 ]; then
printf "Usage:\nbond.sh interface1 interface2 bondname IP boradcastIP netmask network mode link_monitoring_frequency(miimon)\n"
printf "eg: bond.sh eth5 eth6 bond0 192.168.0.100 192.168.0.255 255.255.255.0 192.168.0.0 6 100\n"
exit 0
fi

printf "\t\t*************** Bonded interface script ***************\n"
printf "\t\t*************** Author : Asanga Pradeep ***************\n"  

if [ -f /etc/sysconfig/network-scripts/ifcfg-$1 ]; then

 if [ -f /etc/sysconfig/network-scripts/ifcfg-$2 ]; then
 
  mv /etc/sysconfig/network-scripts/ifcfg-$1 /root/ifcfg-$1
  mv /etc/sysconfig/network-scripts/ifcfg-$2 /root/ifcfg-$2
  echo "Original files moved to /root"
 else
  echo "No "$2" found"
  exit 0
 fi


else
 echo "No "$1" found"
 exit 0
fi

#creating bond ---------------------------------------------------------------

echo "creating bond " $3
touch /etc/sysconfig/network-scripts/ifcfg-$3

echo "DEVICE="$3 >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "BOOTPROTO=static" >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "BROADCAST="$5 >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "IPADDR="$4 >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "IPV6ADDR=" >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "IPV6PREFIX=" >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "IPV6_AUTOCONF=yes" >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "NETMASK="$6 >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "NETWORK="$7 >>  /etc/sysconfig/network-scripts/ifcfg-$3
echo "ONBOOT=yes" >>  /etc/sysconfig/network-scripts/ifcfg-$3

#creating first eth -------------------------------------------------------------------

echo "creating "$1

touch /etc/sysconfig/network-scripts/ifcfg-$1

echo "DEVICE="$1 >> /etc/sysconfig/network-scripts/ifcfg-$1
echo "BOOTPROTO=none" >> /etc/sysconfig/network-scripts/ifcfg-$1
echo "ONBOOT=yes" >> /etc/sysconfig/network-scripts/ifcfg-$1
echo "USERCTL=no"  >> /etc/sysconfig/network-scripts/ifcfg-$1
echo "MASTER=$3" >> /etc/sysconfig/network-scripts/ifcfg-$1
echo "SLAVE=yes" >> /etc/sysconfig/network-scripts/ifcfg-$1
echo "MTU=9000" >> /etc/sysconfig/network-scripts/ifcfg-$1


#creating second eth -------------------------------------------------------------------

echo "creating "$2
touch /etc/sysconfig/network-scripts/ifcfg-$2

echo "DEVICE="$2 >> /etc/sysconfig/network-scripts/ifcfg-$2
echo "BOOTPROTO=none" >> /etc/sysconfig/network-scripts/ifcfg-$2
echo "ONBOOT=yes" >> /etc/sysconfig/network-scripts/ifcfg-$2
echo "USERCTL=no"  >> /etc/sysconfig/network-scripts/ifcfg-$2
echo "MASTER=$3" >> /etc/sysconfig/network-scripts/ifcfg-$2
echo "SLAVE=yes" >> /etc/sysconfig/network-scripts/ifcfg-$2
echo "MTU=9000" >> /etc/sysconfig/network-scripts/ifcfg-$2

#editng modprobe.conf

if [ -f /etc/modprobe.conf ]; then
 
 cp /etc/modprobe.conf /etc/modprobe.conf.bak
 echo "Original file backup as /etc/modprobe.conf.bak"
 echo "alias "$3" bonding" >> /etc/modprobe.conf
 echo "options "$3" mode="$8" miimon="$9 >> /etc/modprobe.conf
 /sbin/modprobe bonding
else
 echo "NO /etc/modprobe.conf found"
fi

Services to disable on Linux

on both RHEL 4 and RHEL 5

chkconfig sendmail off
chkconfig rhnsd off
chkconfig readahead_early off
chkconfig readahead_later off
chkconfig yum-updatesd off
chkconfig cups off
chkconfig iptables off
chkconfig bluetooth off
chkconfig isdn off
chkconfig kudzu off
chkconfig netfs off
chkconfig rpcgssd off
chkconfig rpcidmapd
chkconfig rpcidmapd off
chkconfig auditd off
chkconfig avahi-daemon off
chkconfig firstboot off
chkconfig hidd off
chkconfig portmap off
chkconfig ip6tables off
chkconfig xend off
chkconfig xendomains off
chkconfig autofs off
chkconfig setroubleshoot off 
chkconfig pcscd off 
chkconfig nfslock off 
chkconfig arptables_jf off
chkconfig cups-config-daemon off
chkconfig rawdevices off
chkconfig  readahead off
chkconfig smartd off
chkconfig xinetd off     
chkconfig xfs off
chkconfig pcmcia off

Locking User Accounts

Locking Linux user account

passwd -l {user-name}

-l : This option disables an account by changing the password to a value, which matches no possible encrypted value.

For unlocking the account use:

passwd -u {user-name}

Locking FreeBSD user account

pw lock {username}

unlocking

pw unlock {username}

Locking Solaris UNIX user account

passwd -l {username}

Locking HP-UX user account

passwd -l {username}

For unlocking the HP-UX account you need to edit /etc/passwd file using text editor
vi /etc/passwd

Finding out accounts without password.

Linux display password status

passwd -S {user-name}

Where,
-S : Display account status information. The status information consists of total seven fields. The second field indicates the status of password using following format:

L : if the user account is locked (L)
NP : Account has no password (NP)
P: Account has a usable password (P)

Solaris UNIX display password status
passwd -s {user-name}Where,
-s : Display account status information using following format:

PS : Account has a usable password
LK : User account is locked
NP : Account has no password

OS Watcher

Download information is available on metlink note 301137.1

Data collectors exist for top, vmstat, iostat, mpstat, netstat, ps and an optional collector for tracing private networks. To turn on data collection for private networks the user must create an executable file in the osw directory named private.net. An example of what this file should look like is named Example private.net in the osw directory. This file can be edited and renamed private.net or a new file named private.net can be created. This file contains entries for running the traceroute command to verify RAC private networks.

Start OSW

startOSW.sh interval_in_sec number_of_hours_to_archive_the_data
if not arguments are present then default will be every 10 seconds and 48 hours of data will be archived.
It's a moving window of 48 hours, meaning on the 49th hour the 1st hour's data will be deleted leaving 48 hours worth of data on the system.

Use nohup ./startOSW.sh 60 10 & to start OSW in background.

Stop OSW

to stop the OSW use stopOSW.sh

OSW Output

OSW output is available in OSW_HOME/archive directory. It will have a seperate subdirecotry for each Os utility. Output file will have the following format for its name node_name_OS_utility_YY.MM.DD.HH24.dat

oswiostat

Field Description
r/s :- Shows the number of reads/second
w/s :- Shows the number of writes/second
kr/s :- Shows the number of kilobytes read/second
kw/s :- Shows the number of kilobytes written/second
wait :- Average number of transactions waiting for service (queue length)
actv :- Average number of transactions actively being serviced
wsvc_t :- Average service time in wait queue, in milliseconds
asvc_t :- Average service time of active transactions, in milliseconds
%w :- Percent of time there are transactions waiting for service
%b :- Percent of time the disk is busy device Device name

what to look for

1. Average service times greater than 20msec for long duration.
2. High average wait times.

oswmpstat

Field Description
cpu Processor ID
minf Minor faults
mif Major Faults
xcal Processor cross-calls (when one CPU wakes up another by interrupting it).
intr Interrupts
ithr Interrupts as threads (except clock)
csw Context switches
icsw Involuntary context switches
migr Thread migrations to another processor
smtx Number of times a CPU failed to obtain a mutex
srw Number of times a CPU failed to obtain a read/write lock on the first try
syscl Number of system calls
usr Percentage of CPU cycles spent on user processes
sys Percentage of CPU cycles spent on system processes
wt Percentage of CPU cycles spent waiting on event
idl Percentage of unused CPU cycles or idle time when the CPU is basically doing nothing

what to look for

1. Involuntary context switches (this is probably the more relevant statistic when examining performance issues.)
2. Number of times a CPU failed to obtain a mutex. Values consistently greater than 200 per CPU causes system time to increase.
3. xcal is very important, show processor migration

OSW graph comes bundled with OSW v2.0.0 and higher. Document is available on metalink note 461053.1

It requires java version 1.4 or higher. to use simply run

java -jar -Xmx512M OSWg.jar -i /home/osw/archive

where /home/osw/archive is the path to the direcotry containing the output subdirectories.

For RAC environment to monitor private network copy the provided Exampleprivate.net as private.net into the same directory. Remove all commands in the private.net except for the ones that are matching the current OS. Replace the node names in the private.net with actual node names of the system.

rpm with different architecture

to look up the architecture
rpm -q --queryformat "%{NAME} %{ARCH} \n"


to delete
rpm -e libstdc++.i386
rpm -e libstdc++.x86_64

to update 32 bit versions
up2date --arch=i386 -i compat-db

to update 64 bit versions
up2date --arch=x86_64 -i compat-gcc-32-c++

Meanings of four digits in *Nix permissions

SUID or setuid: change user ID on execution. If setuid bit is set, when a file is executed by a user, the process will have the same rights as the owner of the file being executed.
If set, then replaces "x" in the owner permissions to "s", if owner has execute permissions, or to "S" otherwise.

SGID or setgid: change group ID on execution. Same as above, but inherits rights of the group of the owner of the file on execution. For directories it also may mean that when a new file is created in the directory it will inherit the group of the directory (and not of the user who created the file).
If set, then replaces "x" in the group permissions to "s", if group has execute permissions, or to "S" otherwise.


Sticky bit. It was used to trigger process to "stick" in memory after it is finished, now this usage is obsolete. Currently its use is system dependent and it is mostly used to suppress deletion of the files that belong to other users in the folder where you have "write" access to.
If set, then replaces "x" in the others permissions to "t", if others have execute permissions, or to "T" otherwise.


setuid, setgid, sticky bits are cleared
chmod 0777 test.txt = -rwxrwxrwx test.txt
chmod 0666 test.txt = -rw-rw-rw- test.txt

sticky bit is set
chmod 1777 test.txt = -rwxrwxrwt test.txt
chmod 1666 test.txt = -rw-rw-rwT test.txt

setgid bit is set
chmod 2777 test.txt = -rwxrwsrwx test.txt
chmod 2666 test.txt = -rw-rwSrw- test.txt

setgid and sticky bits are set
chmod 3777 test.txt = -rwxrwsrwt test.txt

setuid bit is set
chmod 4777 test.txt = -rwsrwxrwx test.txt
chmod 4666 test.txt = -rwSrw-rw- test.txt

setuid and sticky bits are set
chmod 5777 test.txt = -rwsrwxrwt test.txt

setuid and setgid bits are set
chmod 6777 test.txt = -rwsrwsrwx test.txt

setuid, setgid, sticky bits are set
chmod 7777 test.txt = -rwsrwsrwt test.txt