The solution is to sync all servers with a NTP server that keep accurate time. For this to work the servers must be able to connect to the internet to access these NTP servers. But for security reason this may not always be possible. However it is possible to have a setup as shown below where only one server connect to the NTP servers and all others get their time sync from this "internal" server without having to connect to the internet.
It is assumed ntp is installed on the Linux servers if not install.
1. Backup the current /etc/ntp.conf file on all servers.
mv /etc/ntp.conf /etc/ntp.conf.orig2. On the server that is going to act as the internal NTP server create a new ntp.conf file (make sure it's owned by root and permissions are 644) in /etc and add the following entries. Important parts are shown in bold
server ntpserver1.org server ntpserver2.org restrict ntpserver1.org mask 255.255.255.255 nomodify notrap noquery restrict ntpserver1.org mask 255.255.255.255 nomodify notrap noquery restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap restrict 127.0.0.1 restrict -6 ::1 # Undisciplined Local Clock. This is a fake driver intended for backup # and when no outside source of synchronized time is available. server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10 driftfile /var/lib/ntp/drift # Key file containing the keys and key identifiers used when operating # with symmetric key cryptography. keys /etc/ntp/keys # Specify the key identifiers which are trusted. #trustedkey 4 8 42 # Specify the key identifier to use with the ntpdc utility. #requestkey 8 # Specify the key identifier to use with the ntpq utility. #controlkey 8ntpserver1/2.org are the NTP servers that are being used for synchronization. (netserver1/2.org used here is just an example).
The restirct line tells what is allowed for the NTP servers, in this case NTP servers are not allowed to modify nor query the local server. The mask 255.255.255.255 limits access to the single IP address of the remote NTP servers.
Third line is the key for allowing other servers to get the time from this server. This is done by removing the noquery line from the restrict and specifiying computers coming from which network segment are allowed to query. In this case all the servers in the local network would be able to get their time sync from this server.
3. Leave the other options as it is.
4. If the server time is lagging far behind the standard time, then use the ntpdate command to get the time updated allowing ntp service to start with a small time lag. To do this first stop the ntp service
/etc/init.d/ntpd stopand then run
ntpdate -u ntpserver1.org 16 Nov 12:58:49 ntpdate[13104]: adjust time server 129.67.1.160 offset -0.070418 secRun the above command 2-3 times.
5. Start the ntp service and monitor the ntp synchronization in progress
/etc/init.d/ntpd start
watch -n 5 ntpq -p
Every 5.0s: ntpq -p Wed Nov 16 13:08:44 2011
remote refid st t when poll reach delay offset jitter
==============================================================================
*ntpserver1.org 192.6.2.82 2 u 7 64 377 3.732 -67.808 0.520
+ntpserver2.org 192.6.2.74 2 u 61 64 377 3.755 -67.672 17.760
LOCAL(0) .LOCL. 10 l 53 64 377 0.000 0.000 0.0016. On other servers (servers get their time sync from the internal NTP server) eg. App and DB servers on the above diagram, create a new /etc/ntp.conf file and add the following entriesserver internal_ntp.server.net restrict internal_ntp.server.net mask 255.255.255.255 nomodify notrap noquery restrict 127.0.0.1 restrict -6 ::1 # Undisciplined Local Clock. This is a fake driver intended for backup # and when no outside source of synchronized time is available. server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10 driftfile /var/lib/ntp/drift # Key file containing the keys and key identifiers used when operating # with symmetric key cryptography. keys /etc/ntp/keys # Specify the key identifiers which are trusted. #trustedkey 4 8 42 # Specify the key identifier to use with the ntpdc utility. #requestkey 8 # Specify the key identifier to use with the ntpq utility. #controlkey 8In this configuraiton the ntp server would be the hostname or the IP of the internal ntp server.
7. If this is a RAC server then before start the ntp service add -x to /etc/sysconfig/ntpd file(metalink note 551704.1)
# Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"8. Start the ntp service and monitor the synchronization progress
Every 5.0s: ntpq -p Wed Nov 16 13:19:40 2011
remote refid st t when poll reach delay offset jitter
==============================================================================
internal_ntp_IP 163.1.2.160 3 u 1 64 1 0.279 0.074 0.001
LOCAL(0) .LOCL. 10 l - 64 0 0.000 0.000 0.001Useful metalink notes
NTP Information and Setup [ID 1010136.1]
Ntpd Does not Use Defined NTP Server [ID 1178614.1]
An Example NTP Client Configuration to use with Oracle Clusterware 11gR2 [ID 1104473.1]
Linux OS Service ntpd [ID 551704.1]
How to Set Up a Network Time Protocol (NTP) Client in Solaris [ID 1005887.1]
CTSSD Runs in Observer Mode Even Though No Time Sync Software is Running [ID 1054006.1]
NTP leap second event causing Oracle Clusterware node reboot [ID 759143.1]
