vSphere: False alarms on high VM Memory usage in vCenter 4.0


image

Since the upgrade to vCenter 4.0 and ESX 4.0 we got a lot of false alarms on VM memory usage. If you take a look in the advanced performance tab, at VM level. You’ll see that the VM is using all the assigned memory. When you take a look on the host OS, you’ll see that there is less memory usage then vCenter reports to you. I asked on Twitter if anyone else had seen this behavior before. @DuncanYB responded with a post, which he did earlier this year.  

So with the information from @DuncanYB I started a search at http://kb.vmware.com/ and found the following KB article: http://kb.vmware.com/kb/1014019. This article describes one of the symptoms that apply on our environment:

Summaries and Symptoms

Issues fixed in this patch (and their relevant symptoms, if applicable) include:

  • Fixes an issue where a guest operating system’s memory usage might be overestimated on Intel systems that support EPT technology or AMD systems that support RVI technology. This issue might cause the memory alarms in vCenter to go off spuriously even if the guest is not actively accessing a lot of memory.
  • Fixes an issue where DVFilter API’s fail for particular message types during message reordering.
  • Fixes an issue where DVfilter socket reads might fail if zero bytes are returned due to a connection close.
  • Fixes an issue with a DVFilter API where ESX might fail if a guest operating system is moved from one vswitch port to another. This fix allows dropping frames which are accidentally or maliciously posted to a different portset.
  • Fixes an issue where incorrect SysAlert() messages might be displayed on certain systems if the number of cache colors is not calculated correctly.
  • Fixes an issue with monitor or vmkernel crashing when running certain guest operating systems with a 32-bit monitor running in binary translation mode.
Deployment Considerations

BEFORE INSTALLING THIS PATCH: If you have set Mem.AllocGuestLargePage to 0 to workaround the high memory usage issue detailed in the Summaries and Symptoms section, undo the workaround by setting Mem.AllocGuestLargePage to 1.

I installed the patch on a Cluster whit this problem. After the installation of the patch mentioned in the KB article above, vCenter keeps sending false alarms. After a short search on the vmtn communities I found the following post of Paul1

I go to the Top-Level in Vcenter, klick "Alarms" and than "Definitions". Edit one of the definitions (don’t change anything) and then save it. After this the old alarms was gone in my environment

After “changing” the VM Memory Alarm definition, vCenter stops sending out false alarms.

PowerCLI: Virtual Machine Disk (VMDK) info v2: Analyze data with Excel


image

In my previous post about this subject I created a script to view the information on the PowerCLI console. With the v2 script you are able to export the data to a CSV file.

$myCol = @()
$vms = get-view -ViewType VirtualMachine  | Where-Object `
{-not $_.config.template} 
foreach($vm in $vms){
    foreach($dev in $vm.config.hardware.Device){
    $MYInfo = "" | Select-Object VMName, DeviceLabel, `
    FileName, DiskMode, ThinProvisioned

       $MYInfo.VMName = $vm.Name

           if($dev.GetType().Name -eq "VirtualDisk"){
                   $MYInfo.DeviceLabel = $dev.DeviceInfo.Label
                $MYInfo.FileName = $dev.Backing.FileName
                $MYInfo.DiskMode = $dev.Backing.DiskMode
                if($dev.Backing.ThinProvisioned){
                $MYInfo.ThinProvisioned = "True"}
                else{$MYInfo.ThinProvisioned = "False"}

                $myCol += $MYInfo
               }
       }
}
$myCol | Export-CSV -NoTypeInformation "D:\scripts\vmdkinfo.csv"

When the script is finished, you can import the CSV file into Excel.  After the import, we can analyze the data with just a simple filter. With a few clicks,  you’re able to view al the VM’s without Thin Provisioned disks.

image

Now we have a list with all the VM’s with Thin Provisioned disks:

image 

So with a small PowerCLI script and the help of Microsoft Excel, you’re able to generate a report with just the information you need. The best part is that it will only cost you couple of minutes of your time 🙂 .

PowerCLI: Virtual Machine Disk (VMDK) info


image

I was creating a small reporting script about the Virtual Machine disk. The things I wanted to report where the File Name (the location of the VMX file), the disk mode (Independent –> Persistent or nonPersistent) and if the disk is Thin Provisioned or not.  But then I thought why reinventing the wheel if Mr PowerCLI LucD has already created such a script. So I started a search on the VMware communities and found a post of @LucD22 which contains the Thin Provisioned “Check”. So I added the items I wanted to see and came to the following script:

get-view -ViewType VirtualMachine  | Where-Object `
{-not $_.config.template} | % {
Write-Host $_.Name -ForegroundColor Yellow
    foreach($dev in $_.config.hardware.Device){
        if($dev.GetType().Name -eq "VirtualDisk"){
            Write-Host "`t" $dev.DeviceInfo.Label = $dev.Backing.FileName
            Write-Host "`t" $dev.DeviceInfo.Label = $dev.Backing.DiskMode
            
            if($dev.Backing.ThinProvisioned){
            Write-Host "`t" $dev.DeviceInfo.Label = "Is Thin Provisioned" `
            -ForegroundColor Green}
            else{
            Write-Host "`t" $dev.DeviceInfo.Label = "Is NOT ThinProvisioned" `
            -ForegroundColor Red }
            } 
    }
}

This is the output of the script:

image

PowerCLI: Set-dvSwitch


image

Last weekend I was playing around with the new dvSwitch feature in vSphere. So I created a dvSwitch and wanted to migrate my VM’s to it. Unfortunately this was not possible with the current version of PowerCLI. Normally you should be able to change the Network switch via:

Get-VM | Get-NetworkAdapter `
| Set-NetworkAdapter -NetworkName "traditional vswitch" -Confirm:$false

There must be a way to do this with PowerCLI but I didn’t know that way. So I asked Luc Dekens and the other PowerCLI guru’s for a solution. A couple of hours later Luc send me a script which was able to do exactly what I wanted to do.

The function / script can be found over here: http://poshcode.org/1373

You can start the function like this: Set-dvSwitch VirtualMachine dvSwitchPortgroup

image

Just wait a couple of seconds till the Reconfigure virtual machine task is ready:

image

You can also run this function against all your VM’s via the following command:

$vms = Get-VM
foreach($vmName in $vms){
    Set-dvSwitch $vmName dvPG_production
}

Just wait a while and all your VM’s are migrated to the new dvSwitch:

image

If you want to start testing with the dvSwitch, keep an eye on http://lucd.info! @LucD22 is going to post an article about what you can do with PowerCLI and the dvSwitch.

PowerCLI: Check CPU/Memory Hot Add


image

In my previous post I created a couple of functions to enable or disable the Hot Add features.  To check these settings, you can run this one-liner:

Get-VM | Get-View | Select Name, `
@{N="CpuHotAddEnabled";E={$_.Config.CpuHotAddEnabled}}, `
@{N="CpuHotRemoveEnabled";E={$_.Config.CpuHotRemoveEnabled}}, `
@{N="MemoryHotAddEnabled";E={$_.Config.MemoryHotAddEnabled}}

The following output will be generated:

image

PowerCLI: Enable/Disable the VM Hot Add features


image

Since the release of vSphere, you are able to Hot Add memory and vCPU. Quote from the VMware website:

Virtual Machine Hot Add Support— The new virtual hardware introduced in ESX/ESXi 4.0 supports hot plug for virtual devices and supports addition of virtual CPUs and memory to a virtual machine without powering off the virtual machine. See the Guest Operating System Installation Guide for the list of operating systems for which this functionality is supported.

 

So I wanted to see, if I was able to enable/disable this settings via PowerCLI and came up with a couple of functions.

The first function enables the Memory Hot Add feature:

Function Enable-MemHotAdd($vm){
    $vmview = Get-vm $vm | Get-View 
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="mem.hotadd"
    $extra.Value="true"
    $vmConfigSpec.extraconfig += $extra

    $vmview.ReconfigVM($vmConfigSpec)
}

You can run the function via the following command:

Enable-MemHotAdd vc01

When you verify the settings in the vSphere Client, You’ll see that the Memory Hot Add feature is enabled.

image

There is only one problem, the setting doesn’t work. You have to shutdown and start the VM, before you are able to hot add memory to the VM. When the VM is started again, I was able to Hot Add extra memory 🙂

image

To add extra memory via PowerCLI, You have to run the following command:

Get-VM -Name "vc01" | Set-VM -MemoryMB "3072"

 

You can use the next function to disable the setting:

Function Disable-MemHotAdd($vm){
    $vmview = Get-VM $vm | Get-View 
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="mem.hotadd"
    $extra.Value="false"
    $vmConfigSpec.extraconfig += $extra

    $vmview.ReconfigVM($vmConfigSpec)
}

 

I have also created two functions which you can use to enable or disable the hot add feature for vCPU’s:

Enable:

Function Enable-vCpuHotAdd($vm){
    $vmview = Get-vm $vm | Get-View 
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="vcpu.hotadd"
    $extra.Value="true"
    $vmConfigSpec.extraconfig += $extra

    $vmview.ReconfigVM($vmConfigSpec)
}

Disable:

Function Disable-vCpuHotAdd($vm){
    $vmview = Get-vm $vm | Get-View 
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="vcpu.hotadd"
    $extra.Value="false"
    $vmConfigSpec.extraconfig += $extra

    $vmview.ReconfigVM($vmConfigSpec)
}

How To: Install VMware Tools on CentOS 5.3


image image

The first step after a fresh install of a CentOS 5.3 server is too install the packages gcc and kernel-devel. When these packages are installed, update them and reboot of the VM:

yum install gcc kernel-devel –y
yum update –y
reboot

Now we have to create a new folder so we can mount the VMware Tools cd-rom.

mkdir /media/cdrom –p

Start the VMware Tools Installer via the GUI:

image

Mount the cd-rom:

mount /dev/cdrom /media/cdrom

create a folder and copy the tar.gz file from the cd-rom to the folder:

mkdir /root/tarz –p
cp /media/cdrom/VMwareTools-4.0.0-xxxxxx.tar.gz /root/tarz/

Open the folder and extract the tar.gz file:

cd /root/tarz
tar zxvf VMwareTools-4.0.0-xxxxxx.tar.gz

Open the vmware-tools-distrib folder and start the installer:

cd vmware-tools-distrib
./vmware-install.pl

You can change the vnic to vmxnet via the following commands:

/etc/init.d/network stop
rmmod pcnet32
rmmod vmxnet
modprobe vmxnet
/etc/init.d/network start

The final step is to reboot the VM.

VMware: Performance Evaluation of VMXNET3 Virtual Network Device


image

VMware published a new paper about the performance evaluation of VMXNET3 virtual network device.

This paper compares the networking performance of VMXNET3 to that of enhanced VMXNET2 (the previous generation of high performance virtual network device) on VMware vSphere 4 to help users understand the performance benefits of migrating to this next‐generation device.

You can find the document here: http://www.vmware.com/resources/techresources/10065

vSphere: You do not have permission to run this command


vmware-logo-new-2009-400-300x48

Since the release of vSphere, there is a new feature called Hardware Status plug-in.

The new vCenter Sever 4 Hardware Status plug-in provides the ability to monitor the hardware health of your VMware ESX hosts,
including key components such as fans, system board, and power supply. The health information displayed by the vCenter Hardware
Status plug-in are defined and provided by the server hardware vendor through the industry-standard Common Information Model
(CIM) interface.

You can find it on a new tab at the host level in vCenter server:

image

But when I want to view the information in the Hardware Status tab I got the error: You do not have permission to run this command

Luckily I was not the only one with this problem and there is already a topic about this error on the VMware Communities: http://communities.vmware.com/message/1360601#1360601

These are the steps I took to fix this issue:

  1. Stop the VMware VirtualCenter Management Webservices service.
  2. Delete the C:\Program Files (x86)\VMware\Infrastructure\tomcat\webapps\vws\data\VcCache-default-0.XhiveDatabase.DB file.
  3. Start the VMware VirtualCenter Management Webservices service.
  4. Reconnect to the vCenter Server
  5. After reconnecting to the vCenter server, I was able to view the Hardware Status again.image

On Page 6 of what-is-new-in-vmware-vcenter-server-4.pdf you can find more information about the Hardware Status tab.

PowerCLI: Export, Import and Create DRS Rules v2


image

In my first post about this subject, I showed a script create by the PowerCLI Master @LucD. A couple of minutes later Hugo Peeters posted a comment why I did not used the object-oriented approach of Powershell. He also posted an example. So in this post I will show you the other way by using Export/Import-CliXml.

First we start with the export part. The following one-liner will export all the DRS Rules to a XML file:

# Export DRS
Get-Cluster -Name "Cluster_01" | Get-DrsRule | `
Export-CliXml 'C:\scripts\drs.xml'

The XML file look like this:

<?xml version="1.0" encoding="utf-16"?>
<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04">
  <Obj RefId="0">
    <TN RefId="0">
      <T>VMware.VimAutomation.Client20.DrsVMAffinityRuleImpl</T>
      <T>System.Object</T>
    </TN>
    <ToString>VMware.VimAutomation.Client20.DrsVMAffinityRuleImpl</ToString>
    <Props>
      <S N="Name">production</S>
      <B N="Enabled">true</B>
      <S N="ClusterId">ClusterComputeResource-domain-c7</S>
      <B N="KeepTogether">true</B>
      <Obj N="VMIds" RefId="1">
        <TN RefId="1">
          <T>System.String[]</T>
          <T>System.Array</T>
          <T>System.Object</T>
        </TN>
        <LST>
          <S>VirtualMachine-vm-23</S>
          <S>VirtualMachine-vm-27</S>
          <S>VirtualMachine-vm-29</S>
          <S>VirtualMachine-vm-33</S>
        </LST>
      </Obj>
    </Props>
  </Obj>
</Objs>

The next one-liner will import the XML file, which we earlier created with the export one-liner. After the import, the one-liner will create a DRS rule:

# Import DRS
ForEach ($rule in (Import-CliXml 'C:\scripts\drs.xml')){
    New-DrsRule -Cluster (Get-Cluster -Name "Cluster_01") `
    -Name $rule.Name -Enabled $rule.Enabled `
    -KeepTogether $rule.KeepTogether `
    -VM (Get-VM -Id $rule.VmIds)}

 

If you want to see the DRS Rules for a cluster via the PowerCLI, You can run the following one-liner:

# DRS Rules One-Liner
Get-DrsRule -Cluster (Get-Cluster -Name Cluster_01) `
| Sort Name | Select Name, @{N="Enabled";E={$_.Enabled}}, `
@{N="KeepTogether";E={$_.KeepTogether}},`
@{N="Virtual Machines";E={$_.VMIds | % { Get-VM -Id $_ | `
% {$_.Name} }}} 

The following output will be generated:

Name Enabled KeepTogether Virtual Machines
appliances False False Nagios, vMA
production True True DC01,VC01,MC01,MAIL01

Before you can create a XML file with your new rules, you need to get the ID’s of the Cluster and the VM. You can get these ID’s via the following one-liner:

Get-VM | Sort Name | Select Name, `
@{N="VM ID";E={$_.Id}}, `
@{N="Cluster Name";E={Get-Cluster -VM $_}}, `
@{N="Cluster ID";E={Get-Cluster -VM $_ | % {$_.Id}}} | fl *

The output will look like this.

image

or you can export it to a CSV file:

Get-VM | Sort Name | Select Name, `
@{N="VM ID";E={$_.Id}}, `
@{N="Cluster Name";E={Get-Cluster -VM $_}}, `
@{N="Cluster ID";E={Get-Cluster -VM $_ | % {$_.Id}}} | `
Export-CSV -NoTypeInformation "D:\scripts\ps\vmids.csv"

 

Now you have to create a XML file. You can use a file, you created with the export DRS one-liner and change the following lines:

image

When you’re done with the XML file, you can run the Import DRS one-liner to import your new Rules. When you open the Cluster properties you will see that the new rule is added:

image