Veeam: Instant Recovery Fails – Unable to Mount filesystem

Before I start with the solution to the error mentioned in the title of this post I want to share some information about Veeam Instant Recovery. I wanted to test the new feature Instant Recovery. So how does Instant Recovery work? This quote comes from the veeam_backup_5_0_user_guide.pdf user guide:

With Veeam Backup & Replication, you can immediately recover a VM from a created backup file. Instant VM recovery accelerates VM restore, allowing you to improve recovery time objectives and decrease downtime of production VMs.

When performing instant recovery, Veeam Backup & Replication creates an independent temporary copy of a VM in your VMware environment and immediately starts it (if necessary). You can then move this copy to your production storage using Storage vMotion and cold migration to finalize recovery, or alternatively, replicate a restored VM with Veeam Backup & Replication and then fail over to the created replica during the next maintenance window. You can also use a recovered VM for testing purposes to make sure VM guest OS and applications are functioning properly.

Similar to the SureBackup recovery verification technology, instant VM recovery does not require you to extract a VM from a backup and move it across datacenter — it mounts a VM directly from a compressed backup file on a selected ESX host. The archived image of a VM remains in a read-only state to avoid unexpected modifications. All changes to a virtual disk that take place while a VM is running are logged to an auxiliary file on the Veeam Backup server or any datastore you select. These changes are discarded as soon as a restored VM is removed.

Let’s start an Instant Recovery restore job:

 

Select the VM you want  to Recover:

Go through the rest of the Wizard and select the destination vsphere host, datastore and resourcepool. When you finished the setup, just wait until the job is ready.

in the vmkernel log on the vSphere host, you will find the following lines:

Oct 25 12:11:48 esx vmkernel: 9:01:27:00.710 cpu0:4107)NFS: 107: Command: (mount) Server: (veeam02) IP: (1.1.1.2) Path: (/VeeamBackup_VEEAM02) Label: (VeeamBackup_VEEAM02) Options: (None)
Oct 25 12:12:19 esx vmkernel: 9:01:27:31.688 cpu0:4107)WARNING: NFS: 905: RPC error 13 (RPC was aborted due to timeout) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server (1.1.1.2)
Oct 25 12:12:21 esx vmkernel: 9:01:27:33.755 cpu3:4107)NFS: 107: Command: (mount) Server: (1.1.1.2) IP: (1.1.1.2) Path: (/VeeamBackup_VEEAM02) Label: (VeeamBackup_VEEAM02) Options: (None)
Oct 25 12:12:52 esx vmkernel: 9:01:28:04.709 cpu2:4107)WARNING: NFS: 905: RPC error 13 (RPC was aborted due to timeout) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server (1.1.1.2)

The recovery task will fail:

Hmm.. Let’s start some basic storage troubleshooting.

• veeam server ping –> vSphere host [Ok]
• vSphere host ping –> veeam  server  [Ok]
• vSphere host vmkping –> veeam server [Failed]
  So the vSphere host cannot vmkping the Veeam server. This means the vmkernel network is unable to reach the Veeam server.  To solve this issue, I added an extra VMKernel network with an IP address in the same subnet of the Veeam server. Now I was able to vmkping to the Veeam Server.

So I started the Instant Recovery wizard again. I didn’t enable the Redirect virtual disk updates option. If you select this option, you’re not able to perform a Storage vMotion.

After finishing the wizard, the recovery will start and the vPower NFS share is mounted successfully.

The VM is published but powered off. So poweron the VM to get it back online. During the startup of the VM I initiated a Storage vMotion. This worked really well:

The VM was back online in less then 30 seconds. That’s amazing right!

Conclusion: Be sure your network setup is ready before you start to deploy Veeam Backup. Be sure the Veeam Backup server is on the same network like the VMkernel network on the vSphere hosts or add a new VMkernel network to be sure Instant Recovery works as expected.