afokkema VMware

Storage vMotion only one hard disk to another datastore in vSphere


Sometimes it’s necessary to only migrate a single hard disk from a VM. This is the case when someone adds two 1 TB VMDK’s and fills them up completely. The maximum size of a VMFS datastore is 2TB minus 512 bytes. So in this case the datastore will be completely filled with no space left to keep the VM running. So if you want  to migrate just one hard disk to make sure the VMFS datastore will not fill up. You can use the vSphere client or PowerCLI to do perform this “advanced” Storage vMotion.

Note: if you want to reclaim your “wasted” storage back from your SAN, you have to recycle the whole datastore. So you have to migrate the other hard disks and configuration files as well.

vSphere Client

Start the Migrate Virtual Machine wizard and select datastore:

image

Continue reading “Storage vMotion only one hard disk to another datastore in vSphere” →

afokkema VMware

vMotion error: Virtual machine must be running in order to be migrated


Today I wanted to Storage vMotion a VM to a new datastore. But for the first time I got a general error message:

image

followed up by a general system error message:

image

I got the same message when I tried to start a “normal” vMotion. So I start to troubleshoot this error. First I looked at the vmware.log of the VM. Nothing unusual in there. So the next stop was the VMkernel. But there was nothing unusual in it too. So I used the good old Microsoft like fix to restart the VMware services at the Service Console using the following command:

service mgmt-vmware restart

after a minute or so I was able to start a vMotion again and after the vMotion completed I started the storage vMotion I was planning to do and this worked like a charm again.

To recap. Sometimes you need to restart the mgmt-vmware to fix the connection between the vSphere host and vCenter.

afokkema VMware

vSphere: Storage vMotion of a virtual machine might stop responding at 18%


For some reason a couple of storage vMotions went really slow. So I looked in the logs and found the following lines in the vmware.log:

Oct 15 11:43:37.889: Worker#0| DISKLIB-VMFS :CopyData ‘/vmfs/volumes/xxxxx/vm1/vm1_1-flat.vmdk’ : failed to move data (Cannot allocate memory:0xc0009).
Oct 15 11:43:37.890: Worker#0| DISKLIB-LIB : DiskLib_Clone: Could not perform clone using vmkernel data mover. Falling back to non-accelerated clone. Cannot allocate memory

I started a Google search and found the following thread http://communities.vmware.com/message/1545132 in the communities. A short quote from richardt his post:

"This problem is caused by the VMFS3-DM (Data Mover) having to use contiguous memory space on the host. Apparently, when host’s kernel memory usage is >50% and memory has been fragmented, the DM cannot allocate more space and throws the errors.

I couldn’t get any fix that day so a made an internal case and got focused on another case. The next day I was reading the release notes of patch ESX400-302009402-SG and found the following quote:

Storage vMotion of a virtual machine might stop responding at 18%, and might be completed after a long time, even though other Storage vMotion operations on the host might continue without any errors. If you try to cancel the Storage vMotion operation when it stops responding, the system disconnects the ESX host from the vCenter Server and automatically connects it after a few minutes.

The vmware.log file might contain the following error:
Could not perform clone using vmkernel data mover. Falling back to non-accelerated clone. Cannot allocate memory

The VMkernel log file might contain the following error:
status Out of memory copying 16777216 bytes from file offset 0 to file offset 0, bytesTransferred = 0 extentsTransferred: 0

This issue occurs because the DataMover cannot allocate contiguous physical space when the host’s kernel memory usage is around 50% and the memory is fragmented. The operation fails back to the Application layer data movement. The operation continues and succeeds, but might take more time when compared to the usual DataMove time. The DataMover requires 16MB of contiguous physical memory on the ESX host for each DataMover thread. This patch provides a fix to make DataMover work with fragmented memory.

Installing patch ESX400-302009402-SG did resolve this issue.

 

Source: KB1023759

afokkema VMware

vSphere: Storage vMotion Fails with a Timeout


image

When I tried to Storage vMotion a VM to another LUN I got an “operation timed out error”:

image

After analyzing the vpxd.log, I found the following lines:

image

The best part I found out, is that the VM was rolled back to it’s old location without losing data or getting corrupt.

This problem is documented in KB1010045. The following solution will resolve the timeout problem:

Increase the Storage VMotion fsr.maxSwitchoverSeconds setting in the virtual machine configuration file to a larger value.

The default is 100 seconds.

To modify fsr.maxSwitchoverSeconds:

  1. Right-click the virtual machine and click Edit Settings.
  2. Click the Options > Advanced > General.
  3. Click Configuration Parameters.
  4. From the Configuration Parameters window, click Add Row.
  5. For the Name field,  fsr.maxSwitchoverSeconds and for the Value field enter the new timeout value.
  6. Click OK twice to save.
  7. Restart the virtual machine for the changes to take effect.

You don’t want to change this by hand, so I created a PowerCLI script which will set the fsr.maxSwitchoverSeconds to 300 seconds for all your VM’s:

$vms = Get-View -ViewType VirtualMachine | where {-not $_.config.template}

    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
      
    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="fsr.maxSwitchoverSeconds"
    $extra.Value="300"

    $vmConfigSpec.extraconfig += $extra
    

foreach($vm in $vms){
    $vm.ReconfigVM($vmConfigSpec)
}

After changing the value to 300 seconds, I was able to Storage vMotion the VM.

afokkema Storage, Virtualization, VMware

VMware: Storage VMotion The Easy Way


 

Sinds ESX 3.5 bestaat de feature Storage VMotion. Deze feature stelt je in staat een actieve VM te verplaatsen naar een andere datastore zonder downtime. Ik was nog niet in de gelegenheid geweest om deze feature te testen. In het begin kon je een Storage VMotion alleen uitvoeren  via de remote cli (deze kun je hier downloaden: http://vmware.com/download/vi/drivers_tools.html) of je kunt gebruik maken van de VIMA. Bij deze twee opties moet je de Storage VMotion taak uitvoeren van de commandline. Gelukkig zijn er altijd nog gasten die kunnen programmeren. Zo ook Andrew Kutz. Hij heeft de Storage VMotion plugin voor VIC gemaakt. Meer informatie en de download vind je hier: http://sourceforge.net/projects/vip-svmotion/.

In de rest van de post lees je hoe eenvoudig het gebruik van deze plugin is.

Continue reading “VMware: Storage VMotion The Easy Way” →