October 1, 2014October 6, 2014 afokkema VMware

HP’s September VMware driver bundle and issues with Emulex CNA’s

Update 08/10/2014: HP support recommends to install the following versions of the Emulex firmware/drivers.

Install the following driver and firmware version for the NIC and install on server and check

As it is a ESX 5.1 we would use BE2NET driver

Driver: 4.9.488.0 – http://www.hp.com/
Firmware: 4.9.416.0 – http://www.hp.com/

You can read more about the issues here at http://www.techazine.com. I saw the same symptoms on brand new HP BL460 Gen8 and vSphere 5.1 update 2, the latest build: VMware-ESXi-5.1.0-Update2-2000251-HP-5.68.30-Sep2014.iso released on 2014-09-08. Hosts randomly disconnects from vCenter, even if those hosts are in maintenance mode (lucky me).

The driver causing the issues is version: net-be2net 10.2.293.0-1OEM.510.0.0.802205 Emulex VMwareCertified 2014-08-25. More information about the drivers included in the latest custom ISO can be found here: http://vibsdepot.hp.com/hpq/recipes/September2014VMwareRecipe16.0.pdf

To solve this issue, you need to downgrade the driver to the previous version. I had no issues with the 4.6.247.10 drivers so I used this one.

I created a PowerCLI script to verify the installed version of the net-be2net on all the BL460c Gen8 blades.

$be2netlist = @()
foreach($vmhost in (Get-VMhost | ?{$_.Model -eq "ProLiant BL460c Gen8"}| sort name)){
    Write-Host "Checking host $($vmhost.name)" -ForegroundColor Yellow
    $esxcli = Get-EsxCli -VMHost $vmhost
        $be2net = New-Object PSObject -Property ([ordered]@{
            vmhost = $vmhost.name
            driver = ($esxcli.software.vib.list() | ?{$_.Name -eq "net-be2net"}).name
            version = ($esxcli.software.vib.list() | ?{$_.Name -eq "net-be2net"}).version        
        })
        $be2netlist += $be2net
}
$be2netlist | ft -AutoSize

The output:

Now we wait for a fix from HP and Emulex.

August 13, 2014 afokkema Veeam

Veeam: End of support Veeam 6.5 in Q4 2014 with the release of v8

A quote from Gostev, in the Veeam Community Forums Digest mailing, send on august 4:

If you, or your customers are still using B&R 6.5, please keep in mind that we will stop supporting this version along with our v8 release, which is planned for early Q4. So please take the opportunity of slow summer time to perform the upgrade to v7. New releases typically increase load on our technical support team, which may in turn increase response times to handle possible upgrade issues (although frankly, we are not getting a lot of those for v7).

So if you’re still using Veeam 6.5 in production. Start working on your RFC to upgrade to Veeam v7.

July 9, 2014July 9, 2014 afokkema Veeam

Veeam v7: How to change the Job notification settings with Powershell

In an earlier post you will find a Powershell script to change the VM Notes setting in Veeam Backup. This old script worked for previous version of Veeam Backup and Replication.

In Veeam Backup v7 the Powershell code is a little bit changed so I had to rewrite the script. The script below will enable the VM notes for a job, if the option is disabled.

if((Get-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue) -eq $null){
    Add-PSSnapin "VeeamPSSnapIn"
}

foreach($vbrjob in (get-vbrjob | Sort Name)){
    $options = $vbrjob.GetOptions()
    if($options.ViSourceOptions.SetResultsToVmNotes -eq $false){
        $options.ViSourceOptions.SetResultsToVmNotes = $true
        Write-Host "Enable set results to VM notes for job $($vbrjob.Name)"    
    }
    $vbrjob.SetOptions($options)
}

June 17, 2014June 24, 2014 afokkema Hardware, VMware

BSOD in Virtual Machines configured with Intel Xeon E5-2x00v2 CPU’s

This article describes the Issue, the servers and BIOS version that are affected by this issue and the BIOS upgrade to fix the issue. I have also added a PowerCLI script to check if your HP servers are affected by this issue. You can find it at the bottom of this article.

Issue:

Random BSOD’s on Windows servers running Windows 2008R2.

On Windows you can see the following blue screen events:
– 0x0000000a – IRQL_NOT_LESS_OR_EQUAL
– 0x0000001a – MEMORY_MANAGEMENT
– 0x000000fc – ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY
– 0x0000004e – PFN_LIST_CORRUPT
– 0x00000050 – PAGE_FAULT_IN_NONPAGED_AREA
– 0x0000003B- SYSTEM_SERVICE_EXCEPTION

VMware has a KB article KB2073791 about this issue. You can find a workaround described there. In he mean time HP and also other vendors released a BIOS update to resolve this issue.

HP Hardware:

HP has release an Advisory document with more info about this issue: c04327904

DESCRIPTION

IMPORTANT : The System ROM update provided in the Resolution is recommended to prevent the issues below from occurring. HP recommends performing this upgrade at the customer’s earliest possible convenience. Neglecting to perform the recommended action and not performing the recommended resolution could result in the potential for subsequent errors to occur.

Intel has identified a processor issue where Virtual Machines (VMs) running on VMware ESX 5.x or Microsoft Hyper-V may experience a Blue Screen of Death (BSOD) in a Windows virtual machine or a kernel panic in a Linux virtual machine when using VMware ESX 5.x or Microsoft Hyper-V. This issue affects Intel Xeon E5-2400 series v2, Intel Xeon E5-2600 series v2, Intel Xeon E5-4600 series v2, Intel Xeon E7-4800 series v2, and Intel Xeon E7-8800 series v2 processors.

Intel has released an updated microcode for impacted processors that addresses this issue. Updated revisions of HP ProLiant System ROMs for impacted servers include this microcode. This issue is NOT unique to HP ProLiant servers and could impact any system utilizing affected processors if the updated microcode is not utilized.

This issue does NOT affect servers configured with Intel Xeon E5-2400 series, Intel Xeon E5-2600 series, or Intel Xeon E5-4600 series processors.

SCOPE

The following servers may be affected if running a System ROM revision dated earlier than indicated when using impacted processors:

HP ProLiant DL160 Gen8- earlier than J03 (02/10/2014 )

HP ProLiant ML350e Gen8 – earlier than J02 (02/10/2014)

HP ProLiant DL380p Gen8- earlier than P70 (02/10/2014 )

HP ProLiant DL360p Gen8 -earlier than P71 (02/10/2014 )

HP ProLiant ML350p Gen8 -earlier than P72 (02/10/2014)

HP ProLiant DL360e Gen8- earlier than P73(02/10/2014 )

HP ProLiant DL380e Gen8-earlier than P73 (02/10/2014 )

HP ProLiant SL230s Gen8- earlier than P75 (02/10/2014 )

HP ProLiant SL250s Gen8- earlier than P75 (02/10/2014 )

HP ProLiant SL270s Gen8-earlier than P75 (02/10/2014 )

HP ProLiant DL560 Gen8- earlier than P77 (02/10/2014 )

HP ProLiant SL4540 Gen8- earlier than P74 (02/10/2014)

HP ProLiant SL210t Gen8- earlier than P83 (02/10/2014 )

HP ProLiant BL420c Gen8- earlier than I30 (02/10/2014 )

HP ProLiant BL460c Gen8- earlier than I31 (02/10/2014 )

HP ProLiant BL660c Gen8- earlier than I32 (02/10/2014 )

HP ProLiant DL580 Gen8- earlier than P79 (04/01/2014 )

Affected Intel Processors supported by the above servers:

Intel Xeon Processor E5-2400 Series v2

Intel Xeon Processor E5-2600 Series v2

Intel Xeon Processor E5-4600 Series v2

Intel Xeon Processor E7-4800 Series v2

Intel Xeon Processor E7-8800 Series v2

RESOLUTION

This issue is resolved by updating the System ROM to the versions indicated as follows:

HP ProLiant DL160 Gen8 – J03 (02/10/2014 or later)

HP ProLiant ML350e Gen8 – J02 (02/10/2014 or later)

HP ProLiant DL380p Gen8 – P70 (02/10/2014 or later)

HP ProLiant DL360p Gen8 – P71 (02/10/2014 or later)

HP ProLiant ML350p Gen8 – P72 (02/10/2014 or later)

HP ProLiant DL360e Gen8 – P73(02/10/2014 or later)

HP ProLiant DL380e Gen8 – P73 (02/10/2014 or later)

HP ProLiant SL230s Gen8 – P75 (02/10/2014 or later)

HP ProLiant SL250s Gen8 – P75 (02/10/2014 or later)

HP ProLiant SL270s Gen8 – P75 (02/10/2014 or later)

HP ProLiant DL560 Gen8 – P77 (02/10/2014 or later)

HP ProLiant SL4540 Gen8 – P74 (02/10/2014 or later)

HP ProLiant SL210t Gen8 – P83 (02/10/2014 or later)

HP ProLiant BL420c Gen8 – I30 (02/10/2014 or later)

HP ProLiant BL460c Gen8 – I31 (02/10/2014 or later)

HP ProLiant BL660c Gen8 – I32 (02/10/2014 or later)

HP ProLiant DL580 Gen8 – P79 (04/01/2014 or later)

Download the BIOS upgrade for HP Servers and Blades here:

DL380 gen8: http://www.hp.com
DL360 Gen8: http://www.hp.com
BL460 Gen8: http://www.hp.com

http://itnervecenter.com/content/windows-bugchecks-vmware-esxi-xeon-e5-2670-cpus

PowerCLI script:

update: PowerCLI script: compare dates fixed.

I created a script to see if the hosts are affected by this issue or not. The script will collect all ESXi hosts with HP hardware and the V2 Xeon processor installed. All those hosts are checked by Model, for instance DL360 Gen8. If the Model matches, the script will check the release date of the BIOS and returns a line of text in Red if the release date is older than the one mentioned in the HP Advisory: c04327904 or else a line of text in Green if the release date matches or newer than the one mentioned in the Advisory document.

foreach($vmhost in (Get-VMHost | Where{$_.ProcessorType -match "v2"} | Sort Name)){
    $hpModel = $vmhost.Model.TrimStart("ProLiant ")

    switch ($hpModel){
    "DL160 Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "ML350e Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "DL380p Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "DL360p Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "ML350p Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "DL360e Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "DL380e Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "SL230s Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "SL250s Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "SL270s Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "DL560 Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "SL4540 Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "SL210t Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "BL420c Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "BL460c Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
                }
    "BL660c Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    "DL580 Gen8" {
                    if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"04/01/2014")){
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green
                    }
                    else{
                        Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red
                    }
    
                 }
    default {"The server model could not be determined."}
    }
}

The output of the script above:

January 3, 2014January 3, 2014 afokkema VMware

vShield Host preparation: Host not reachable

Just a quick tip when you’re using vShield and want to prepare a new ESXi host. When you click on the vShield tab inside the vSphere client you might see the message “Host not reachable”.

So I started with some basic troubleshooting like ping from the vShield Manager to the ESXi host and from the ESXi host to the vShield Manager. This worked. The next step was the debug connection command from the vShield Manager appliance. This command completed successfully:

So what was the issue?? Well it’s simple the ESXi host was in Maintenance mode while trying to perform the Host preparation. This is not supported. The host needs to be in normal operation mode to be able to connect to vShield. Now I was able to prepare the ESX host:

November 7, 2013 afokkema VMware

Syslog gone mad after vSphere upgrade to vSphere 5.1 update 1

After upgrading a vSphere 5.0 update 2 host to vSphere 5.1 update 1 we noticed an issue with the lsassd daemon. Right after update manager finished with the upgrade the lsassd starts to write a lot of messages. Within the last 5 minutes the syslog server received 170K log messages from the upgraded host alone.

See the screenshot below:

The following message keeps popping up in the logs:

esxihost.domain.loc lsassd[9297]: 0x6eb11b90:Terminating on fatal IPC exception

To work around this issue you need to leave the Windows domain. Select the host – Configuration – Authentication Services – Properties. Click on Leave Domain… The excessive logging hast stopped immediately.

After that you can rejoin to the Windows domain again.

September 19, 2013September 19, 2013 afokkema VMware

Import OVF/OVA: No datastore found on target in vCenter Server 5.1

I was trying to import an OVF/OVA template to one of my lab environments. But unfortunately this was not possible because of an error:

In this environment we are using Folders to organize the datastores. See the screenshot below:

This works for most of the time but when you want to import an OVF/OVA template you have to give up your folder structure to enable the OVF template. This is a small bug in vCenter Server 5.1. See KB2045635 for more details.

This is a small quote from the KB article:

This is a known issue affecting VMware vCenter Server 5.1.

Try one of the following workarounds:

Use vSphere Client to deploy.
Open the vSphere Client and use it to deploy the OVA/OVF file.

Use Web Client to deploy.
If the datastore is in a folder, remove it from the folder and try to deploy.

In this case I had to move the SDRS cluster and the datastores attached to this SDRS cluster from the yellow folder to the root folder inside the Storage view of the vSphere web client.

Now I was able to deploy the OVF/OVA template.

July 19, 2013 afokkema Veeam

Veeam: A file level restore gets stuck in a "stopping" state.

Today a colleague of mine was trying to perform a file level restore on one of the Veeam servers. But unfortunately the job became unresponsive and got stuck in a stopping state. See the print screen below:

To fix this issue, you need to run a SQL query on the Veeam database to clear the locks. I have included the solution form KB1534:

Problem:

A file level restore gets stuck in a "stopping" or "starting" state.

Cause:

This issue typically occurs when the VeeamBackup database has become out of sync with Veeam and the true state of the File Level Restore is not updated in the GUI.

Solution:

To fix this you will need to run a SQL query against the VeeamBackup Database, instructions are as shown below.
Before running this SQL query against the “VeeamBackup” Database, please make sure that there are currently no jobs running. If you are unaware of how to run a query against a DB please read the steps below, if you are aware please disregard.
1.       Open up SQL Server Management Studio and connect to the VEEAM instance.
2.       Expand “Databases”.
3.       Right-click “VeeamBackup”>New query.
4.       Paste the query below into the query window and hit either “execute” or F5.

The query:

UPDATE [ReportRestoreSessionsAndTaskSessionsView]
   SET "state" = -1 
    WHERE "initiator_name" not like 'null'

After running the query and restarting the Veeam console. My colleague was able to perform the file level restore.

Source: http://www.veeam.com/kb1534

May 2, 2013July 8, 2013 afokkema VMware

PowerCLI: enable SSH and configure ESXi Firewall

It’s a long time ago when I posted a new article on my blog so it’s time to write some new content. I want to start with a post about configuring SSH to start automatically, hide the Shell warning message and configure the ESXi firewall to allow the connection from a certain IP address. Of course all this is done by running a PowerCLI script.

But first I want to show you where you can change the ESXi firewall settings. Go to the configuration tab and select the Security Profile. Select the rule you want to change and click on firewall.. Select the option “Only allow connections from the following networks” and add the IP address or IP range you want to allow.

But like I mentioned before this is not a job do by hand when you have a large vSphere environment so I want to share the PowerCLI script below to perform this task for you. The only things you need to change are the $cluster and $ip variables. Then copy the script to your PowerCLI session and run it.

$cluster = "<clusterName>"
$ip = "192.168.1.1"

foreach($vmHost in (Get-Cluster $cluster | Get-VMHost | Sort Name)){
    write-host "Configuring SSH on host: $($vmHost.Name)" -fore Yellow
    if((Get-VMHostService -VMHost $vmHost | where {$_.Key -eq "TSM-SSH"}).Policy -ne "on"){
        Write-Host "Setting SSH service policy to automatic on $($vmHost.Name)"
        Get-VMHostService -VMHost $vmHost | where { $_.key -eq "TSM-SSH" } | Set-VMHostService -Policy "On" -Confirm:$false -ea 1 | Out-null
    }

    if((Get-VMHostService -VMHost $vmHost | where {$_.Key -eq "TSM-SSH"}).Running -ne $true){
        Write-Host "Starting SSH service on $($vmHost.Name)"
        Start-VMHostService -HostService (Get-VMHost $vmHost | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"}) | Out-null
    }    
    
    $esxcli = Get-EsxCli -VMHost $vmHost
    if($esxcli -ne $null){
        if(($esxcli.network.firewall.ruleset.allowedip.list("sshServer") | select AllowedIPAddresses).AllowedIPAddresses -eq "All"){
            Write-Host "Changing the sshServer firewall configuration"        
            $esxcli.network.firewall.ruleset.set($false, $true, "sshServer")
            $esxcli.network.firewall.ruleset.allowedip.add("$ip", "sshServer")
            $esxcli.network.firewall.refresh()
        }    
    }
    
    if(($vmHost | Get-AdvancedSetting | Where {$_.Name -eq "UserVars.SuppressShellWarning"}).Value -ne "1"){
        Write-Host "Suppress the SSH warning message"
        $vmHost | Get-AdvancedSetting | Where {$_.Name -eq "UserVars.SuppressShellWarning"} | Set-AdvancedSetting -Value "1" -Confirm:$false | Out-null
    }    
}

The script checks if the SSH Service is running or not and will change the setting is necessary. This is also the case with the Firewall configuration and the part to suppress the Shell warning message.

March 22, 2013June 6, 2013 afokkema VMware

PowerCLI: Automatic expand the available ports for a dvPortgroup

Last year William Lam wrote a blog post on the VMware vSphere Blog about automatic expand a dvPort group port. You can find his post here: http://blogs.vmware.com/vsphere/. There is also a KB article about this subject. You can find it here: KB1022312

Just a quote from the KB article KB1022312 to explain the auto expand feature and how you can enable it without using Perl of PowerCLI scripting:

Note: vSphere 5.0 has introduced a new advanced option for static port binding called Auto Expand. This port group property allows a port group to expand automatically by a small predefined margin whenever the port group is about to run out of ports. In vSphere 5.1, the Auto Expand feature is enabled by default.

In vSphere 5.0 Auto Expand is disabled by default. To enable it, use the vSphere 5.0 SDK via the managed object browser (MOB):

In a browser, enter the address http://vc-ip-address/mob/.

When prompted, enter your vCenter Server username and password.

Click the Content link.

In the left pane, search for the row with the word rootFolder.

Open the link in the right pane of the row. The link should be similar to group-d1 (Datacenters).

In the left pane, search for the row with the word childEntity. In the right pane, you see a list of datacenter links.

Click the datacenter link in which the vDS is defined.

In the left pane, search for the row with the word networkFolder and open the link in the right pane. The link should be similar to group-n123 (network).

In the left pane, search for the row with the word childEntity. You see a list of vDS and distributed port group links in the right pane.

Click the distributed port group for which you want to change this property.

In the left pane, search for the row with the word config and click the link in the right pane.

In the left pane, search for the row with the word autoExpand. It is usually the first row.

Note the corresponding value displayed in the right pane. The value should be false by default.

In the left pane, search for the row with the word configVersion. The value should be 1 if it has not been modified.

Note the corresponding value displayed in the right pane as it is needed later.

Go back to the distributed port group page.

Click the link that reads ReconfigureDvs_Task. A new window appears.

In the Spec text field, enter this text:
<spec> <configVersion>1</configVersion> <autoExpand>true</autoExpand> </spec>
where configVersion is what you recorded in step 15.

Click the Invoke Method link.

Close the window.

Repeat Steps 10 through 14 to verify the new value for autoExpand.

If you need to change this setting for hundreds of dvPortgroups this will not be one of your favorite changes in your VMware environment. Well you know me. Let’s see if we can PowerCLI this job.

$dvPG = Get-VirtualPortGroup -Name "VM Network"
$dvPGview = get-view $dvPG
$spec = New-Object VMware.Vim.DVPortgroupConfigSpec
$spec.AutoExpand = "True"
$spec.ConfigVersion = $dvPGview.Config.ConfigVersion
$dvPGview.ReconfigureDVPortgroup_Task($spec)
$dvPGview.UpdateViewData()

if you want to change all the dvPortgroups at one. You can use the following script:

Update: Thanks to Rafael Schitz from http://www.hypervisor.fr/ for the tip to filter out the dvUplink Portgroups. I have also added a check to find out if the dvSwitch is running the correct version to enable the autoExpand feature. Copy the script below and change de $dvSwitchName variable to the name of your dvSwitch.

$dvSwitchName = "dvSwitchName"
$dvSwitch = Get-VirtualSwitch -Distributed -Name $dvSwitchName
if($dvSwitch.ExtensionData.Config.ProductInfo.Version –notmatch "4.*"){
    foreach($dvPG in (Get-View -ViewType DistributedVirtualPortgroup|?{!($_.Tag|?{$_.Key -eq "SYSTEM/DVS.UPLINKPG"}) -and !$_.Config.autoExpand})){
        $spec = New-Object VMware.Vim.DVPortgroupConfigSpec
        $spec.AutoExpand = "True"
        $spec.ConfigVersion = $dvPG.Config.ConfigVersion
        $dvPG.ReconfigureDVPortgroup_Task($spec)
        Write-Host "Enable auotExpand for dvPortgroup: $($dvPG.Name)" -ForegroundColor Yellow
        $dvPG.UpdateViewData()
    }
}
else{
    Write-Host "dvSwitch: $($dvSwitch.Name) is not configured with version 5 or higher. Please upgrade.." -ForegroundColor Red
}

If the dvPortgroup has 0 available ports and a VM wants to connect a network adapter to the dvPortgroup, The total ports variable will be automatically expand with 10 ports. After a couple of tests I can confirm that it works.

My VM Network dvPortgroup had 70 ports. When al these ports where claimed by VM’s and a new VM was deployed or an existing VM was configured with a new network adapter, the Available ports variable was expanded with 10 ports without any impact for the running VM’s.