vSphere: VM “freezes” during the removal of a snapshot

If your VM is running on a NFS datastore and Changed Block Tracking (CBT) is enabled, you might experience the following symptoms:

When removing the snapshot of the virtual machine residing on the NFS storage in an ESX/ESXi 4.1 host, you may experience these symptoms:

  • The virtual machine appears to be in a hung state within the console view
  • The virtual machine stops responding to ping requests
  • The virtual machine is inaccessible via remote desktop

But why is this VM freezing? The explanation is the locking mechanism which NFS uses:

This issue may occur if you are using a backup application that utilizes Changed Block Tracking (CBT) and the ctkEnabled option for the virtual machine is set to true. The virtual machine becomes unresponsive for up to 30 seconds as NFS locks on the .ctk file are changed.

The workaround is to disable CBT:

  1. Wait for the virtual machine to become responsive and for the snapshot operation to complete.
  2. Schedule an outage window for the affected virtual machine.
  3. Shut down the virtual machine.
  4. In the Inventory pane, right-click the virtual machine and click Edit Settings.
  5. Click the Options tab and click General.
  6. Click Configuration Parameters.
  7. If the ctkEnabled  parameter is not listed, click Add Row, add ctkEnabled, and set it to false.
  8. Power on the virtual machine.

This workaround can impact the backup performance because you can’t use CBT. To Disable CBT in a Veeam job, you have to edit the job and disable CBT in the Advanced Settings under the vSphere tab:


Or if you’re using PHD Virtual Backup you can change the CBT settings in the Options tab on the properties page of a Backup Job:



http://kb.vmware.com KB1031106

Review: PHD Virtual Backup

Before I start with this post I first want to welcome a new sponsor: PHD Virtual.


In this post I will be doing a short review, sponsored by PHD, of the latest version of PHD backup. In this post I will show the following topics:

  1. install the PHD Virtual Backup Console and Plug-in
  2. Deploy OVF template of the PHD Virtual Backup Appliance
  3. Configure the PHD Virtual Backup Appliance
  4. Add VMs to a Backup job
  5. Restore a complete VM
  6. Restore individual Files
  7. Conclusion

So let’s start with the installation.

Continue reading “Review: PHD Virtual Backup”

Powershell: Script to Query the Veeam Backup database


Earlier this year I wrote a post about how to query the Veeam Backup SQL database to get the total job running time. I wanted to see if I was able to run this Query via Powershell. So I started to search on Google and I found a great series of articles on http://www.databasejournal.com about how to use Powershell to access Microsoft SQL databases. After reading part two, I was able to create a script to run my Query.

The only thing you have to change are the next three variables:

$dbServer = "servername\instance" 
$db = "VeeamBackup"
$veeamJob = "VeeamJobName"


Run the next script to query the Veeam Backup database and return the total job time.

$dbServer = "servername\instance"
$db = "VeeamBackup"
$veeamJob = "VeeamJobName"
$Query = "SELECT [job_name],CONVERT(char(10),[creation_time], 101) AS start_date `
,CONVERT(varchar, [creation_time], 108) AS job_start,CONVERT(char(10), [end_time], 101) AS end_date `
,CONVERT(varchar, [end_time], 108) AS job_end, `
LEFT(CONVERT(VARCHAR,CAST([end_time] AS DATETIME)-CAST([creation_time] AS DATETIME), 108),5) AS total_time `
FROM [VeeamBackup].[dbo].[BSessions] WHERE [job_name] = '$veeamJob' ORDER BY start_date"

$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=$dbServer;Database=$db;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $Query
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$DataSet.Tables[0] | Format-Table -AutoSize 

You can also find the script on poshcode.org: http://poshcode.org/1316


The script generate the following output:


Release: Veeam Backup


@Gostev has published the news on Twitter:


The Release Notes:

New Features
• Added support for replication and remote backup over slow and unreliable links with moderate packet loss.
Resolved Issues
• .NET Framework bug affecting some non-English locales (confirmed to exist on Norwegian locale) results in sporadic corruption of backup files chain (missing VBK files, extra VRB files).
• Backup file repair after unsuccessful backup cycle fails if backup storage is low on free disk space.
• Network backup in service console agent mode may fail with the following error: “Failed to validate command "dd if="/vmfs/volumes/…”.
• Attempting to enable SSH connection to ESX host on Japanese locale fails with the following error: ["Could not find any resources appropriate for the specified culture or the neutral culture. Make sure "VeeamSSH.strings.ja.resources" was correctly embedded or linked into assembly "Veeam.SSH" at compile time, or that all the satellite assemblies required are loadable and fully signed."].
• Veeam VSS processing fails on specific OS configurations with various “Access denied” errors while specified VSS account has suffient permissions.
• Veeam VSS processing fails on specific OS configurations with the following error: "The specified service does not exist as an installed service".
• Veeam VSS integration does not support guests with Datacenter Edition of Microsoft Windows 2003.

You can download the new release here: http://www.veeam.com/vmware-esx-backup/download.html

Source: http://www.veeam.com/forums/viewtopic.php?f=2&t=1565

Veeam Backup: Not enough licenses

Nadat ik dit weekend de ESX Servers opnieuw had geïnstalleerd (upgrade naar ESX 3.5u3) werkte Veeam Backup niet meer. De volgende foutmelding verscheen in de logfiles:


Dit is eenvoudig op te lossen. Ga naar Help –> License Information…


Klik op Licensed hosts


Klik daarna op de ESX Server gevolgd door Revoke


Daarna moet de backup weer normaal lopen.

Proven practices for integrating Backup Software with VCB


Chris Skinner een Senior Technical Instructor at VMware PSO-Education heeft een aantal papers geschreven over het integreren van backup software en VCB.

Op de viops site van VMware kun je de volgende papers vinden:



Bron: http://www.ntpro.nl/blog/archives/692-Integrating-Symantec-Backup-Exec-12.5-with-VCB-1.5.html

Whitepaper: VMware and VSS: Application Backup and Recovery

Veeam schrijft het volgende:

More and more organizations are choosing VMware Infrastructure to virtualize their mission‐critical applications (Active Directory, Exchange, SharePoint, SQL Server) to create a flexible, easily administered virtual infrastructure.
Virtual machines (VMs) and any applications they contain must be protected against failure. Typically, in the virtual world, this is done by performing an image‐level backup of the whole machine (for instance, using VMware Consolidated Backup). This method results in what is known as a crash‐consistent image. Restoring a crash‐consistent image is essentially equivalent to rebooting a server after a hard reset. For operating systems, this has not been an issue, since they can easily handle this type of activity. For database applications as well as for applications featuring replication, however, such a restore will often result in lost data, data corruption, or application failure.
To illustrate this concept, let’s review the consequences of using different VM disaster recovery methods for one of the most common mission‐critical applications: a Microsoft Active Directory domain controller (DC). Depending on the solution used, the following results will be achieved:


Download de PDF hier: VMwareandVSS-ApplicationBackupandRecovery.pdf

Backup Exec: Snapshot provider error (0x8007000E): Ran out of mem

Sinds ik SP2 op de SQL server had geïnstalleerd, kreeg ik de onderstaande foutmelding in mijn Backup Exec log file.

Click an error below to locate it in the job log

Backup- SQL1 
AOFO: Initialization failure on:
Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

Snapshot provider error (0x8007000E): Ran out of memoryCheck the Windows Event Viewer for details.

Dit probleem kun je oplossen door de hotfix te bestellen die vermeldt word in het volgende KB document: KB940239

Symantec: Backup Exec and an error with Shadow Copy Components

De onderstaande foutmelding kreeg ik toen ik een SQL server wilde toevoegen in mijn backup list van Symantec Backup Exec 10d.


Meer informatie over deze foutmelding vind je hier: http://seer.entsupport.symantec.com/docs/291014.htm


Deze foutmelding kun je oplossen door de volgende dll opnieuw te registreren ole32.dll

Dit doe je als volgt: Start – Run – regsrv32 ole32.dll {Enter}


De oplossing kwam uit een post van Zhu [MSFT] welke je hier kunt vinden: http://forums.techarena.in/showthread.php?t=498777

Unattended: BackupExec 10d Remote Agent

In de map C:\Program Files\VERITAS\Backup Exec\NT\Agents\RANT32 op de Backup Exec server staan de benodigde installatie bestanden. Deze kopieer je naar je installatie directory.

In de installatie directory vindt je de onderstaande cmd files.

  • setupaa.cmd de standaard Remote Agent
  • setupafo.cmd de Remote Agent met de Advanced Open File Option

In het setupaa.cmd bestand vindt je het volgende commando: @start /wait setup /RANT32: -s

In het setupafo.cmd bestand vindt je het volgende commando: @start /wait setup /AOFO: -s

Zie deze technote 258982 voor meer informatie.