This article describes the Issue, the servers and BIOS version that are affected by this issue and the BIOS upgrade to fix the issue. I have also added a PowerCLI script to check if your HP servers are affected by this issue. You can find it at the bottom of this article.
Issue:
Random BSOD’s on Windows servers running Windows 2008R2.
On Windows you can see the following blue screen events:
– 0x0000000a – IRQL_NOT_LESS_OR_EQUAL
– 0x0000001a – MEMORY_MANAGEMENT
– 0x000000fc – ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY
– 0x0000004e – PFN_LIST_CORRUPT
– 0x00000050 – PAGE_FAULT_IN_NONPAGED_AREA
– 0x0000003B- SYSTEM_SERVICE_EXCEPTION
VMware has a KB article KB2073791 about this issue. You can find a workaround described there. In he mean time HP and also other vendors released a BIOS update to resolve this issue.
HP Hardware:
HP has release an Advisory document with more info about this issue: c04327904
DESCRIPTION
IMPORTANT : The System ROM update provided in the Resolution is recommended to prevent the issues below from occurring. HP recommends performing this upgrade at the customer’s earliest possible convenience. Neglecting to perform the recommended action and not performing the recommended resolution could result in the potential for subsequent errors to occur.
Intel has identified a processor issue where Virtual Machines (VMs) running on VMware ESX 5.x or Microsoft Hyper-V may experience a Blue Screen of Death (BSOD) in a Windows virtual machine or a kernel panic in a Linux virtual machine when using VMware ESX 5.x or Microsoft Hyper-V. This issue affects Intel Xeon E5-2400 series v2, Intel Xeon E5-2600 series v2, Intel Xeon E5-4600 series v2, Intel Xeon E7-4800 series v2, and Intel Xeon E7-8800 series v2 processors.
Intel has released an updated microcode for impacted processors that addresses this issue. Updated revisions of HP ProLiant System ROMs for impacted servers include this microcode. This issue is NOT unique to HP ProLiant servers and could impact any system utilizing affected processors if the updated microcode is not utilized.
This issue does NOT affect servers configured with Intel Xeon E5-2400 series, Intel Xeon E5-2600 series, or Intel Xeon E5-4600 series processors.
SCOPE
The following servers may be affected if running a System ROM revision dated earlier than indicated when using impacted processors:
- HP ProLiant DL160 Gen8- earlier than J03 (02/10/2014 )
- HP ProLiant ML350e Gen8 – earlier than J02 (02/10/2014)
- HP ProLiant DL380p Gen8- earlier than P70 (02/10/2014 )
- HP ProLiant DL360p Gen8 -earlier than P71 (02/10/2014 )
- HP ProLiant ML350p Gen8 -earlier than P72 (02/10/2014)
- HP ProLiant DL360e Gen8- earlier than P73(02/10/2014 )
- HP ProLiant DL380e Gen8-earlier than P73 (02/10/2014 )
- HP ProLiant SL230s Gen8- earlier than P75 (02/10/2014 )
- HP ProLiant SL250s Gen8- earlier than P75 (02/10/2014 )
- HP ProLiant SL270s Gen8-earlier than P75 (02/10/2014 )
- HP ProLiant DL560 Gen8- earlier than P77 (02/10/2014 )
- HP ProLiant SL4540 Gen8- earlier than P74 (02/10/2014)
- HP ProLiant SL210t Gen8- earlier than P83 (02/10/2014 )
- HP ProLiant BL420c Gen8- earlier than I30 (02/10/2014 )
- HP ProLiant BL460c Gen8- earlier than I31 (02/10/2014 )
- HP ProLiant BL660c Gen8- earlier than I32 (02/10/2014 )
- HP ProLiant DL580 Gen8- earlier than P79 (04/01/2014 )
Affected Intel Processors supported by the above servers:
- Intel Xeon Processor E5-2400 Series v2
- Intel Xeon Processor E5-2600 Series v2
- Intel Xeon Processor E5-4600 Series v2
- Intel Xeon Processor E7-4800 Series v2
- Intel Xeon Processor E7-8800 Series v2
RESOLUTION
This issue is resolved by updating the System ROM to the versions indicated as follows:
- HP ProLiant DL160 Gen8 – J03 (02/10/2014 or later)
- HP ProLiant ML350e Gen8 – J02 (02/10/2014 or later)
- HP ProLiant DL380p Gen8 – P70 (02/10/2014 or later)
- HP ProLiant DL360p Gen8 – P71 (02/10/2014 or later)
- HP ProLiant ML350p Gen8 – P72 (02/10/2014 or later)
- HP ProLiant DL360e Gen8 – P73(02/10/2014 or later)
- HP ProLiant DL380e Gen8 – P73 (02/10/2014 or later)
- HP ProLiant SL230s Gen8 – P75 (02/10/2014 or later)
- HP ProLiant SL250s Gen8 – P75 (02/10/2014 or later)
- HP ProLiant SL270s Gen8 – P75 (02/10/2014 or later)
- HP ProLiant DL560 Gen8 – P77 (02/10/2014 or later)
- HP ProLiant SL4540 Gen8 – P74 (02/10/2014 or later)
- HP ProLiant SL210t Gen8 – P83 (02/10/2014 or later)
- HP ProLiant BL420c Gen8 – I30 (02/10/2014 or later)
- HP ProLiant BL460c Gen8 – I31 (02/10/2014 or later)
- HP ProLiant BL660c Gen8 – I32 (02/10/2014 or later)
- HP ProLiant DL580 Gen8 – P79 (04/01/2014 or later)
Download the BIOS upgrade for HP Servers and Blades here:
- DL380 gen8: http://www.hp.com
- DL360 Gen8: http://www.hp.com
- BL460 Gen8: http://www.hp.com
- But what if you’re using Dell hardware. Well please take a look at http://itnervecenter.com/content/windows-bugchecks-vmware-esxi-xeon-e5-2670-cpus for more information about issue and the fix.
PowerCLI script:
update: PowerCLI script: compare dates fixed.
I created a script to see if the hosts are affected by this issue or not. The script will collect all ESXi hosts with HP hardware and the V2 Xeon processor installed. All those hosts are checked by Model, for instance DL360 Gen8. If the Model matches, the script will check the release date of the BIOS and returns a line of text in Red if the release date is older than the one mentioned in the HP Advisory: c04327904 or else a line of text in Green if the release date matches or newer than the one mentioned in the Advisory document.
foreach($vmhost in (Get-VMHost | Where{$_.ProcessorType -match "v2"} | Sort Name)){ $hpModel = $vmhost.Model.TrimStart("ProLiant ") switch ($hpModel){ "DL160 Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "ML350e Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "DL380p Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "DL360p Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "ML350p Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "DL360e Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "DL380e Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "SL230s Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "SL250s Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "SL270s Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "DL560 Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "SL4540 Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "SL210t Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "BL420c Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "BL460c Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "BL660c Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"02/10/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } "DL580 Gen8" { if((Get-Date $vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -ge $([datetime]"04/01/2014")){ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Green } else{ Write-Host "vmHost: $($vmHost.Name.split(".")[0]) Model: $($vmhost.Model.TrimStart("ProLiant ")) BIOS version: $($vmhost.ExtensionData.Hardware.BiosInfo.BiosVersion) Release date $(((Get-Date ($vmhost.ExtensionData.Hardware.BiosInfo.ReleaseDate) -format MM-dd-yyyy).Replace("-","/")))" -for Red } } default {"The server model could not be determined."} } }
The output of the script above:
This is a very useful article. Full range of HP only in Senetic.