Linux: Disk Timeout settings not increased by VMware Tools

Recently I had some issues with Linux VM’s which became read-only. In my earlier post about disk-timeout settings I wrote about the timeout value being increased during the VMware Tools installation. But how does the VMware Tools install change this value. I though the solution can be found within the vmware-config-tools.pl script. So to find the vmware-config-tools.pl script just run:

[root@linuxvm1 ~]# type vmware-config-tools.pl
vmware-config-tools.pl is /usr/bin/vmware-config-tools.pl

No run the less commmand:

less /usr/bin/vmware-config-tools.pl

press / and type 180 now you see the info we are looking for:

The disk timeout value can only be changed with Linux kernel 2.6.13 or higher. Ok so what if you use a Linux distribution with a kernel older than 2.6.13? From KB51306:

VMware has identified a problem wherein file systems may become read-only after encountering busy I/O retry or SAN or iSCSI path failover errors.

The same behavior is expected even on a native Linux environment, where the time required for the file system to become read-only depends on the number of paths available to a particular target, the multi-path software installed on the operating system, and whether the failing I/O was to an EXT3 Journal. However, the problem is aggravated in an ESX Server environment because ESX Server manages multiple paths to the storage target and provides a single path to the guest operating system, which effectively reduces the number of retries done by the guest operating system.

These guest operating systems are affected:

• RHEL5 (RedHat)
• RHEL4 U6
• RHEL4 U4
• RHEL4 U3
• SLES10
• SLES9 SP3 
  Note: This issue may affect other Linux distributions based on early 2.6 kernels as well, such as Ubuntu 7.04.

This situation can lead to serious issues and can only be solved with a reboot of the VM. But there is a workaround. From KB1009465:

Increasing the timeout value

The timeout value for a Linux block device can be set using sysfs.
Note: This is usually increased automatically when deploying VMware-Tools, but if it is not installed, you will need increase it manually.

Check the current values using the following command:

for a in /sys/class/scsi_generic/*/device/timeout; do echo -n "$a "; cat "$a" ; done;

Increase the timeout value for an individual disk using the following command. For example to change the values for device sdc, run:

echo 180 > /sys/block/sdc/device/timeout

Run the following command to change the timeout values for all devices to 180:

for i in /sys/class/scsi_generic/*/device/timeout; do echo 180 > "$i"; done

you can add the following command:

for i in /sys/class/scsi_generic/*/device/timeout; do echo 180 > "$i"; done

to the /etc/rc.d/rc.local file to make sure the disk timeout is changed during startup.

Source:
KB1009465	http://kb.vmware.com/
KB51306	http://kb.vmware.com/
VMTN communities	http://communities.vmware.com/thread/257251