Grid Control is calling  >> “Difference between OMS system time and Agent system time is 121 mins and has exceeded the critical threshold 120 mins”

The system time runs too fast on a (Suse) Linux-based (kernel 2.6) virtual machine, while the ntp has been configured. There are a lot of documents regarding this kind of issues. For explanation and understanding a few documents we used:
http://www.vmware.com/pdf/vmware_timekeeping.pdf
http://support.microsoft.com/kb/918461 (Microsoft? seems to work for VM-ware too…)
http://www.novell.com/coolsolutions/feature/15345.html (time not syncing with ntp-server)

Next option for me when all above should not be working, but didn’t have to use it:
http://ict-freak.nl/2007/01/10/how-to-setup-ntp-on-vmware-esx-server/

Solution we implemented, seemed to work:
– add the clock=pit parameter to the kernel entry (for explanation, see below)
– in addition add the option ‘burst iburst‘ to the ntp-configuration.

To see by the way how the ntp is working:
# watch ntpq -p

Clock=pit for the GRUB bootloader
In the guest operating system, open the /boot/grub/menu.lst file by using a text editor such as Vi. For example, type the following command from a console, and then press ENTER:
vi /boot/grub/menu.lst
This file contains the Linux boot options and resembles the following:
title Linux
kernel (hd0,4)/vmlinuz root=/dev/hda7 vga=791
initrd (hd0,4)/initrd
title windows
etc.
In the title Linux area of this file, add the clock=pit parameter to the kernel entry. This area should resemble the following:
title Linux
kernel (hd0,4)/vmlinuz root=/dev/hda7 vga=791 clock=pit
initrd (hd0,4)/initrd
Save the changes to the file, exit Vi, and then restart the Linux-based virtual machine.

Clock=pit for the LILO bootloader
In the guest operating system, open the /etc/lilo.conf file by using a text editor such as Vi. For example, type the following command from a console, and then press ENTER:
vi /etc/lilo.confThis file contains the Linux boot options and resembles the following:

### LILO Linux section (default)
image  = /boot/vmlinuz     # Default
label  = linux
root   = /dev/hda7         # Root partition for the kernel
initrd = /boot/initrd

In the ### LILO Linux section (default) area of this file, type the following entry:
Append = “clock=pit”
This area should resemble the following:
### LILO Linux section (default)
image  = /boot/vmlinuz     # Default
label  = linux
root   = /dev/hda7         # Root partition for the kernel
initrd = /boot/initrd
Append = “clock=pit”
Save the changes to the file, and then exit Vi.
Run the lilo command. For example, type the following command, and then press ENTER:
/sbin/lilo
Restart the Linux-based virtual machine.

For a better understanding:

After you install a Linux-based virtual machine in Microsoft Virtual Server 2005 R2, the system time in the Linux guest operating system runs too fast. For example, the current time in the Linux guest operating system may advance by one minute approximately every 48 seconds. After some time, the Linux-based virtual machine resets the clock to the correct time.

You experience this issue if you are running a Linux distribution that uses the Linux 2.6 kernel.  Some theoretical explanation about the time synchronization and kernel versions:
Time synchronization in a virtual machine
An operating system generally tracks time by using the periodic time interrupts that are generated by a specific hardware device. Generally, an operating system obtains the time from a battery-backed Complimentary Metal Oxide Semi-conductor (CMOS) clock during the operating system’s startup procedure. The operating system then configures a timer device to generate periodic interrupts. The operating system keeps track of time by counting these interrupts.

For a virtual machine, the actual physical hardware is shared by the host operating system and by the guest operating system. When a virtual machine generates a time interrupt, the guest operating system may be running or may not be running. Therefore, the guest operating system does not immediately account for some of these interrupts. To work around this issue, the virtual machine keeps a backlog of these interrupts. Additionally, the virtual machine increases the frequency of timer interrupts when it is running. The increased frequency of timer interrupts is intended to help the guest operating system maintain the correct time. However, the increased frequency of these interrupts could cause the guest operating system to miss some of the interrupts. These missed interrupts are known as “lost ticks.” Lost ticks cause the guest operating system time to lag behind the actual time. Although you may experience this issue on a physical computer, you are more likely to experience this issue in a guest operating system that is running on a virtual machine.

The Linux 2.4 kernel and earlier Linux kernels rely on the timer interrupts that are delivered by the timer. The algorithms that are implemented for time synchronization in the Linux 2.4 kernel do not account for lost ticks. This behavior may cause the Linux guest operating system time to lag behind the actual clock. To help resolve this issue, some Linux distributions patch the Linux 2.4 kernel to deliver timer interrupts at a faster rate. However, the Linux guest operating system could still experience time synchronization issues because of lost ticks.

The Linux 2.6 kernel implements more efficient algorithms to resolve this time synchronization issue. Unlike the Linux 2.4 algorithms, the Linux 2.6 algorithms adjust for lost ticks. However, this adjustment may cause the Linux-based virtual machine to gain time. The Linux 2.6 kernel has three different clock parameters that can be passed to the kernel at boot time. Use these parameters to select the algorithms to use for time synchronization. For more information about these algorithms, see the “Timekeeping in Linux” section. Each of these timekeeping algorithms has advantages and disadvantages. However, you are more likely to notice disadvantages on a virtual machine than on a physical computer.