VMware: BUG in ESX 3.5 Update 2

De Bug:

Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error “A general system error occurred: Internal Error”.
Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:
Aug 12 10:40:10.792: vmx| http://msg.License.product.expired This product has expired.
Aug 12 10:40:10.792: vmx| Be sure that your host machine’s date and time are set correctly.
Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: http://www.vmware.com/info?id=4.

De workaround:

The work-around: turn off NTP (if you’re using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s “08/10/2008” at the Service Console command line on the ESX hosts.

As soon as the date was reset to the 10th – problem solved.

Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.

Bron: http://communities.vmware.com/thread/162377?tstart=0

Reactie van een medewerker van VMware:

“Dear VMware customers,

We are actively working on rootcausing the problem. Once we know the appropriate action to take here, we’ll provide an update.

Apologies for any inconvenience.

The ESX Product Team”

Bron: http://vmetc.com/2008/08/11/august-12-bug-in-esx-35-update-2/

Op het VMUG forum staat nu ook een topic: http://www.vmug.nl/modules.php?name=Forums&file=viewtopic&t=2953

Daarin schrijft JCvanDoorn het volgende:

Hier is een kort lijstje met wat recommendations van een van mijn collega’s:
1. Set DRS to manual
2. Do not VMotion
3. Do not power off or suspend VM
4. If you have some important VMs you need to turn on again
a. Create a quarantine host – a standalone host without any VMs
b. Set time on this host back
c. Verify, that your VM will not synchronize its time with that host
d. Cold migrate VM to the quarantine host
e. Power VM on, log into it and set check the time setting. If necessary set the time manually
You could also think about temporarily disconnecting the VM to power on from the Virtual Switch. This way you are certain that the VM will not be able to communicate with a DC while it’s out of time-synch. This allows you to set the time manually before connecting it back to the Virtual Switch.

Inmiddels heeft VMware zelf ook een melding op hun supportsite geplaatst: http://www.vmware.com/support/

En het topic in de VMware communities is nog steeds groeiende:  http://communities.vmware.com/message/1020565