PowerCLI: Copy Datastore Items

In this short post I will show a PowerCLI script I wrote to copy ISO files from datastore y to datastore x. The datastores are in the same vCenter and virtual datacenter accessible but the vSphere hosts are located inside two different IP subnets and a firewall rule prevents to copy files between the two subnets. So I had to think about a work around. Well this one is easy. On the vCenter server I created a script to peform the following steps:

  1. Create two PSDrives for each Datastore
  2. Get al the ISO filenames
  3. Downlad the ISO to the c:\tmp directory from datastore y
  4. Upload the ISO from the C:\tmp directory to the datastore<X>\iso directory
  5. Remove the ISO from C:\tmp
  6. repeat the steps above until all the ISO files are copied to the new datastore.

The PowerCLI script to perform the described tasks:

New-PSDrive -location (get-datastore template-01) -name tpl01 -PSProvider VimDatastore -Root '\'
New-PSDrive -location (get-datastore template-02) -name tpl02 -PSProvider VimDatastore -Root '\'

$isos = ls tpl01:\iso\ | % {$_.Name}
foreach($iso in $isos){
    Write-Host "copy $($iso) to C:\tmp" -fore Yellow
    Copy-DatastoreItem -item tpl01:\iso\$iso -Destination C:\tmp
    Write-Host "copy $($iso) to template-02\iso" -fore Yellow
    Copy-DatastoreItem -item C:\tmp\$iso -Destination tpl02:\iso
    Write-Host "removing the tmp file $($iso) from C:\tmp" -fore Yellow
    Remove-Item C:\tmp\$iso -confirm:$false
    Write-Host "done" -fore Green

So once again PowerCLI to the rescue.

Error 25114. Setup failed to generate the JRE SSL keys

Today I was busy with a vCenter server upgrade to vCenter 4.1 update 2. Everything went fine except the vCenter Update manager installation. I received the following error:


The solution is pretty simple this time. Just be sure to stop the vCenter Update Manager service before starting the setup. Right after stopping the service, the installation was successful and I was happy again. In the VMware Communities you’ll find that this issue is also know for the upgrade of vCenter 5 to vCenter 5 update 1. See http://communities.vmware.com/ for more info. Now let’s patch some vSphere hosts with the help of PowerCLI: powercli-update-vmhost-function/

vCOPS 5: HTTP Status 404 –

When I opend the vCOPs v5 page this morning a got the following error:


So I checked if the services where running at the time. You can do this with the vcops-admin status command:


The next thing to check is the free disk space with the df –h command:


The disk seems to be full. To fix this issue, You can follow the steps from KB2016645:

To add a new virtual disk to the virtual machine:

  1. Power off the vApp.
  2. In the vSphere Client, right-click the virtual machine and click Edit Settings.
  3. Add the additional virtual disk.
    Note: Ensure to consider the future growth while selecting the disk size.
  4. Power on the vApp. The virtual machine automatically configures the newly added disk at boot time.

When the VM starts the disk is added to the VM and the LVM logical group will be configured with the new disk:


And we’re back:


PowerCLI: Disable / Enable HA Host Monitoring

In the case you need to or your network team needs to do some network maintenance on the switches which VMware HA uses to communicate with the other hosts or where the das.isolationaddress (default gateway) is configured/ It’s smart to disable the Host Monitoring feature of VMware HA. You can do this easily by hand via edit cluster – VMware HA and uncheck the Enable Host Monitoring feature. See screenshot below:


But what if you have to disable Host Monitoring on multiple VMware HA cluster? Well, if you like PowerCLI, you can use the following script to disable or enable the HA Host Monitoring feature:


if($vCenter -eq $null){
    Write-Host "Please enter the name of the vCenter Server" -ForegroundColor Yellow

    enabled {"The HA Host Monitoring feature will be enabled"}
    disabled {"The HA Host Monitoring feature will be disabled"}
    default {"the option value could not be determined."

Connect-VIServer $vCenter

$clspec = New-Object VMware.Vim.ClusterConfigSpecEx
$clspec.dasConfig = New-Object VMware.Vim.ClusterDasConfigInfo
$clspec.dasConfig.hostMonitoring = $option

foreach($cluster in (Get-Cluster | sort Name)){
    $clview = Get-Cluster $cluster | Get-View
    $clview.ReconfigureComputeResource_Task($clspec, $true)

Disconnect-VIServer -Confirm:$false

Just save the script to change-HAHostMonitoring.ps1 and run it like this to disable the HA Host Monitoring feature:

Change-HAHostMonitoring vcenter.domain.loc disabled

If you want to enable Host Monitoring, just change disabled to enabled:

Change-HAHostMonitoring vcenter.domain.loc enabled

Note: Please test the script mentioned in this blog post in a lab or test environment before you use the script in a production environment.

NMP: nmpDeviceAttemptFailover: Retry world failover device

After restarting a vSphere 4.1 update 2 host there where a lot of warnings in the vmkernel log about NMP and failover messages:

Feb 29 16:03:19 esx vmkernel: 0:00:53:08.502 cpu1:4305)WARNING: NMP: nmpDeviceAttemptFailover: Retry world restore device "mpx.vmhba34:C0:T0:L0" - no more commands to retry
Feb 29 16:03:24 esx vmkernel: 0:00:53:13.498 cpu0:4096)VMNIX: VmkDev: 2860: abort succeeded.
Feb 29 16:03:24 esx vmkernel: 0:00:53:13.498 cpu0:4096)WARNING: NMP: nmp_IssueCommandToDevice: I/O could not be issued to device "mpx.vmhba34:C0:T0:L0" due to Not found
Feb 29 16:03:24 esx vmkernel: 0:00:53:13.498 cpu0:4096)WARNING: NMP: nmp_DeviceRetryCommand: Device "mpx.vmhba34:C0:T0:L0": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
Feb 29 16:03:24 esx vmkernel: 0:00:53:13.498 cpu0:4096)WARNING: NMP: nmp_DeviceStartLoop: NMP Device "mpx.vmhba34:C0:T0:L0" is blocked. Not starting I/O from device.
Feb 29 16:03:25 esx vmkernel: 0:00:53:14.500 cpu1:4305)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "mpx.vmhba34:C0:T0:L0" - issuing command 0x41027fa7e340
Feb 29 16:03:25 esx vmkernel: 0:00:53:14.500 cpu1:4305)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "mpx.vmhba34:C0:T0:L0" - failed to issue command due to Not found (APD), try again...
Feb 29 16:03:25 esx vmkernel: 0:00:53:14.500 cpu1:4305)WARNING: NMP: nmpDeviceAttemptFailover: Logical device "mpx.vmhba34:C0:T0:L0": awaiting fast path state update...
Feb 29 16:03:34 esx vmkernel: 0:00:53:23.500 cpu0:4096)VMNIX: VmkDev: 2767: a/r=2 cmd=0x1e sn=4054 dsk=vml0:88:0 reqbuf=0000000000000000 (sg=0)
Feb 29 16:03:34 esx vmkernel: 0:00:53:23.500 cpu15:4127)ScsiDeviceIO: 1688: Command 0x1e to device "mpx.vmhba34:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Feb 29 16:03:34 esx vmkernel: 0:00:53:23.500 cpu15:4127)WARNING: NMP: nmp_DeviceStartLoop: NMP Device "mpx.vmhba34:C0:T0:L0" is blocked. Not starting I/O from device.
Feb 29 16:03:34 esx vmkernel: 0:00:53:23.500 cpu0:4096)VMNIX: VmkDev: 2812: abort sn=4054, vmkret=0.

There’s only one thing weird about these warnings because the vmhba mentioned in the vmkernel log isn’t visible inside the vSphere client:


So to verify what kind of device the vmhba was I logged in on the vSphere host and ran the esxcfg-scsidevs –l command:

[root@esx ~]# esxcfg-scsidevs -l | grep vmhba34
   Display Name: Local USB Direct-Access (mpx.vmhba34:C0:T0:L0)
   Devfs Path: /vmfs/devices/disks/mpx.vmhba34:C0:T0:L0

This particular vSphere host was a Dell R710 server with the Dell OMSA agent installed. After a quick search on http://kb.vmware.com I found the following KB article KB1013818 which describes the following cause:


This issue occurs if you have external USB devices and the iDRAC is set to mount these devices as virtual media.

The fix is quite simple. Just detach the iDRAC virtual CD drive or run the following commands to fix the “issue”:

# cd /opt/dell/srvadmin/sbin
# mv invcol invcol.bak
# srvadmin-services.sh restart

After restarting the services the VMkernel logs are clean again.

Source:  KB1013818

PowerCLI: Migrate templates during the Enter Maintenance Mode task

Normally when you put a host into Maintenance mode the templates will stay on the host instead of being migrate to a different host. This can be very annoying if you are performing maintenance on the vSphere host and a colleague needs to deploy a VM from the template. I am running vSphere 4.1 update 1. I don’t know if this is still the case with vSphere 5. The host in Maintenance mode will look like this:


So to fix this annoying “issue” I have created a PowerCLI function to place the vSphere host into maintenance mode and if there are Templates registered on the vSphere host, the Templates will be moved to another host in the Cluster.

Function Enter-MaintenanceMode{
.SYNOPSIS   Enter Maintenance mode 
.DESCRIPTION   The function starts the Enter Maintenance task and also migrates the Templates to another host.
.NOTES   Author:  Arne Fokkema
   One vmHosts.
   PS> Enter-MaintenanceMode<vmHost Name>
  PS> Get-VMHost <vmHost Name> | Enter-MaintenanceMode

    [parameter(ValueFromPipeline = $true,
    position = 0,
    Mandatory = $true,
    HelpMessage = "Enter the vmHost to start the Enter Maintenance mode task")]

    $templates = Get-VMHost $vmHost | Get-Template
    if($templates -eq $null){
        $tplMigrate = $false
        $tplMigrate = $true
    $targetVMHost = Get-VMHost -Location (Get-Cluster -VMHost (Get-VMhost $vmHost)).Name | Where {$_.Name -ne $vmHost} | Sort Name | Select -First 1
    if($tplMigrate -eq $true){
        foreach($tpl in $templates){
            Write-Host "Converting template $($tpl.Name) to VM" -ForegroundColor Yellow
            $vm = Set-Template -Template (Get-Template $tpl) -ToVM 
            Write-Host "Moving template $($tpl.Name) to vmHost: $($targetVMHost)" -ForegroundColor Yellow
            Move-VM -VM $vm -Destination (Get-VMHost $targetVMHost) -Confirm:$false | Out-Null
            Write-Host "Converting template $($tpl.Name) back to template" -ForegroundColor Yellow
            ($vm | Get-View).MarkAsTemplate() | Out-Null    
    Write-Host "Enter Maintenance mode $($vmHost)" -ForegroundColor Yellow
    Set-VMHost $vmHost -State Maintenance | Out-Null

You can run the script like this:

Enter-MaintenanceMode esx07

Or from the pipeline:

Get-VMHost esx07 | Enter-MaintenanceMode

The output will be the same:


And the host is completely empty and ready for maintenance:


Troubleshoot TCP connection issues from the ESX Service Console

If you need to troubleshoot TCP connection issues from your ESX host. You will notice that Telnet isn’t available on your ESX host. But VMware posted a workaround in KB1010747. The reason why VMware did not include the Telnet package is simple:

The TELNET package does not ship with the ESX service console. While the TELNET daemon is clearly a security risk, the TELNET client may be a useful tool in diagnosing TCP session connectivity between the ESX Service Console and TCP ports on a foreign host.

The workaround is a Python script. You can copy the following script to your ESX host and place it in /tmp directory.

# TCP connection "tester" -
# provide the hostname/IP address followed by port number (if no port
# is specified, 23 is assumed ;)
# program will connect to the port and read till either it receives a
# newline or 5 seconds expire
# be sure to chmod 755
import sys
import telnetlib
import socket

PORT = ""
argc = len(sys.argv)

if argc == 3 :
    PORT = sys.argv[2]
elif argc < 2 or argc > 3:
    print "usage %s host <port> \n" % sys.argv[0]

HOST = sys.argv[1]

    tn = telnetlib.Telnet(HOST,PORT)
except socket.error, (errno, strerror):
    print " SockerError( %s ) %s\n" %  (errno, strerror)

print tn.read_until("\n ",5)
print "connection succeeded\n"

If you want to test if a particular TCP port is reachable from your ESX host. You can use the script like this:

[root@esx01 ~]# ./testtcp vc01.ict-freak.loc 443

connection succeeded

[root@esx01 ~]#

In case the TCP port is not reachable from your ESX host. The script will just hang on your command and will eventually time out:

[root@esx01 ~]# ./testtcp vc01.ict-freak.loc 443
SockerError( 110 ) Connection timed out

You can also cancel the script by pressing CTRL + C

I didn’t test the script from ESXi. If you did, please leave a comment.

Source: KB1010747

Disconnect ISO files from Templates with PowerCLI

Just a quick post about how to disconnect ISO files from templates with PowerCLI.  With the following script you can set the CD Drive to No Media. So the ISO files will be disconnected from the Template VMs.

$templates = Get-Template 
foreach($tpl in $templates){
    $vm = Set-Template -Template (Get-Template $tpl) -ToVM 
    Get-CDDrive -VM $vm | Set-CDDrive -NoMedia -Confirm:$false | Out-Null
    ($vm | Get-View).MarkAsTemplate() | Out-Null

First the script will fill the templates variable with all the templates available. The next step is to convert the Template back to a VM. When the template is converted to a VM the Get-CDDrive cmdlet is used to set the CD Drive to No Media. When the CD Drive is configured the VM will be converted back to a template. In stead of nine mouse clicks per template you can lean back and drink your cup of coffee or thee and see the magic powered by PowerCLI.

How to copy ISO files between datastores via PowerCLI

We decided to replace our current central ISO datastore to a new one on different storage. You can use the CP commend on the Service Console, if you’re still running ESX classic. But I wanted to investigate if there was a way to do this with PowerCLI. Well there is cmdlet for that called Copy-DatastoreItem. In the PowerCLI help you will find something like this:


Synopsis Copies items between datastores and between a datastore and a local file system provider.

Syntax Copy-DatastoreItem [-Item] [[-Destination]] [-Force] [-PassThru] [-Recurse] [-WhatIf] [-Confirm]

But how do you find the datastore where you want to copy files from?  Well you need to keep in mind the following two objects. The Folder, if you created one (1) and the Datacenter where the Datastore belongs to (2):


Now that you know the path to the datastore containing the files you want to copy. You can start PowerCLI and connect to your vCenter server. Before you continue, you need to create a temp directory to temporarily save the ISO files in. In my case this is E:\iso. The one-liner will look like this:

Copy-DatastoreItem vmstore:\Alkmaar\DataCenter\template-01\iso\* E:\iso

And now we wait until the files are copied to the temp directory:


The next step is to copy the files from the temp directory to the new datastore.

Copy-DatastoreItem E:\iso\* vmstore:\DataCenter\template-01\iso\

And we wait again:


View Composer: Error during provisioning: Failed to authenticate to AD

Last weekend I was busy with a vCenter migration from vCenter 4.0 on a 32 bit Windows 2003 server to a vCenter 4.1 update 1 server on a Windows 2008 R2 64 bit VM. The database is already running on a separate database server.  The one thing special in this setup was VMware View Composer. So I started the migration and everything went well. vCenter was up and running and VMware View Composer service was started. So it was time to test the provisioning of new Desktops. I changed the pool to deploy 2 new desktops. The process ended with an error:image

I was unable to fix this by myself so I contacted VMware Support and after a while we came up with the following solution: Login to the VMware View Administrator and browse to the vCenter Server page:View Configuration – Servers and select the vCenter server in the the vCenter Servers window and press edit. Now select the Quickprep user for the Desktop pool with the error and press edit:


And enter the password for the Quickprep user:


After re-entering the password for the Quickprep user, we where ready to test the Desktop pools again. We did a test with an existing Desktop pool and a new pool and both worked as expected. The Desktop pools are working again. I want to thank VMware support for the quick and accurate support.