Avamar 7.0.1.61 Patch for “Host is in a state that cannot perform snapshot operations” after upgrade to vCenter 5.5

If you are experiencing the following VMware backup failures after upgrading vCenter to version 5.5,

“VM Image backups that fail due to the Host in a state that cannot perform snapshot operations.”

 I would recommend contacting EMC Support and requesting a hot fix for upgrading your proxy version to AvamarVMwareCombined-linux-sles11sp3-x86_64-7.0.101-62.rpm

ISSUES FIXED:

—————–

Bug 59992 – CLONE: bug 52705–esc 7270:VM image backup fail due to the Host <esxserver> is in a state that cannot perform snapshot operations

Bug 60379 – CLONE: bug 58837–esc 7122:vm image backups show completed successfully but log shows they failed

Bug 60499 – esc 7345:HOTFIX:Hermosa SP1 proxy are NOT clearing stale migration reservation

Bug 60758 – CLONE: bug 59963–esc 7265:Prior snapshots are preventing backup

Bug 60759 – CLONE: Esc 6951: Certain cancellations can cause a cascade of asserts that appears to have no end [FIX – This hotfix allows avtar to terminate quickly when canceled in this situation]

Bug 60998 – CLONE: bug 58218-backup VM does not fail back to unquiesce when quiesce snapshot failed

Data Domain OS version 5.3 encouraged to upgrade to 5.4

I recommend customers running Data Domain with operating system (OS)  5.3 to upgrade to OS 5.4 as soon as possible for support purposes.  The upgrade will assist in quicker time to resolution and more efficient troubleshooting from EMC Customer Service and EMC Data Domain Engineering.  As always, please review the DD OS release notes, especially the Known Issues and Workarounds section before proceeding. 

Avamar 7 – Cleaning up mounted snapshots, hidden snapshot and orphaned files when receiving “createSnapshot: snapshot creation failed” errors

The following are some steps for cleaning up mounted snapshots, hidden snapshot and orphaned files when Avamar VMware Image backups fail with the following error, “createSnapshot: snapshot creation failed”.  

You can edit the properties of the Avamar Proxy appliance in vCenter and if you see more than 2 vmdk’s present then could use the following steps to clean up orphaned snapshots. Please note that proxcp.jar is a powerful tool and I would highly recommend engaging EMC Support or an EMC Partner for assistance when using it. You may also want to keep it located in the /home/admin directory.  

 

• Putty into the Avamar Utility Node as root and download the proxcp.jar file to the /tmp directory:

               cd /tmp

                curl -O ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/proxycp.jar 

• To see a list of commands, run the following from the Avamar Utility Node:

               java -jar proxycp.jar

• To view VM’s with hidden Snapshot Files, run this command:

              java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Snapshot

              Output example from my lab:

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

 

• To remove original disk from the proxy that they were still mounted to, use this command.

                  java -jar proxycp.jar –listproxydisk –cleanup

                 Output example from my lab:

VDP/Nemo Detected : false

————————————————————————————

Listing single instace of a Proxy, use –all to list all the instances

====================== grif-vcenter.whughgriffin.com ============================

——————————————————————————-

Proxy Name     Grif-Proxy                         (grif-proxy-proxy-1)               

——————————————————————————-

Agent Version  : 7.0.100-427                      

Disk Key       : 2000                             

Attached Disk  : [VMFS_LOCAL] Grif-Proxy/Grif-Proxy.vmdk

Disk Key       : 2018                             

Attached Disk  : [VMFS_LOCAL] Grif-Proxy/Grif-Proxy_1.vmdk

——————————————————————————————————————————————–

Closing Connection Pool : April 3, 2014 9:31:28 AM EDT

 

• Next clean up Avamar only snapshots with this command if any are present

                  java -jar proxycp.jar –listsnapshots –cleanup

                  Output example from my lab:

root@grif-ave:/tmp/#: java -jar proxycp.jar –listsnapshots –cleanup

======================= Starting ProxyCP ==========================

 

Proxycp : v1.89

Date : April 3, 2014 1:29:14 PM EDT

 

COMMAND : java -jar proxycp.jar –listsnapshots –cleanup 

 

VDP/Nemo Detected : false

————————————————————————————

VM Name(Avamar)                    : Grif-DC                       

VM Name(Vmware)                    : Grif-DC                       

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_NFS] Grif-DC_1/Grif-DC.vmdk

Thin Provisioned                   : true                          

Total Size (GB)                    : 40                            

—————————————————————————————-

VM Name(Avamar)                    : Grif-Desktop                  

VM Name(Vmware)                    : Grif-Desktop                  

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop.vmdk

Thin Provisioned                   : true                          

Total Size (GB)                    : 100                           

—————————————————————————————-

VM Name(Avamar)                    : Grif-Webmail                  

VM Name(Vmware)                    : Grif-Webmail                  

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail.vmdk

Thin Provisioned                   : false                         

Total Size (GB)                    : 40                            

—————————————————————————————-

VM Name(Avamar)                    : Grif-Webmail2                 

VM Name(Vmware)                    : Grif-Webmail2                 

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

Thin Provisioned                   : false                         

Total Size (GB)                    : 40                            

Hidden Snapshot File               : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

—————————————————————————————-

VM Name(Avamar)                    : Grif-Win8                     

VM Name(Vmware)                    : Grif-Win8                     

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-Win8/Grif-Win8.vmdk

Thin Provisioned                   : false                         

Total Size (GB)                    : 100                           

—————————————————————————————-

 

• Check for orphaned snapshots causing exit code 10056

                      java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Hidden

                      Output example from my lab:

root@grif-ave:/tmp/#: java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Hidden

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

 

 

• Using the above list you may proceed using the command below to attempt to consolidate the vm’s that are failing.  Before proceeding I would check with the VMware Administrative staff and update them on the issue, you/they may be able to address the issue with a simple Storage vMotion to a different VMFS datastore and running the backup again.  The VM snapshots may also be consolidated via vCenter using the vSphere Client.  Once again, consolidating VM snapshots could have irreversible affects on the VM.  If you are not comfortable with this step, I would urge you to open a Service Ticket with VMware Support and EMC Avamar Support.

         java -jar proxycp.jar –consolidatesnapshots –vm REPLACEWITHVMNAME

         Output example from my lab:

root@grif-ave:/tmp/#: java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail

======================= Starting ProxyCP ==========================

 

Proxycp : v1.89

Date : April 3, 2014 9:33:34 AM EDT

 

COMMAND : java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail 

 

VDP/Nemo Detected : false

————————————————————————————

It is recommended that you run ‘–listproxydisk –cleanup’ option before consolildation, Please ‘CTRL C’ to exit, waiting 10 seconds

^Croot@grif-ave:/tmp/#: java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail

======================= Starting ProxyCP ==========================

 

Proxycp : v1.89

Date : April 3, 2014 9:35:14 AM EDT

 

COMMAND : java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail 

 

VDP/Nemo Detected : false

————————————————————————————

It is recommended that you run ‘–listproxydisk –cleanup’ option before consolildation, Please ‘CTRL C’ to exit, waiting 10 seconds

VM Name(Avamar)                    : Grif-Webmail                  

VM Name(Vmware)                    : Grif-Webmail                  

Snapshot Name                      : No Snapshots                  

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk

 

Consolidating Snaphot by “Taking new Snapshot & Deleting All Snapshot” !!

Please be patient, It might take a while !!!

 

101% ################################################## /

Snapshot “Avamar-Proxycp” was created.

 

Removing All the snapshots,  Please wait !!

101% ################################################## /

Removed All Snapshots : Time taken : 1 Min 44 Sec 538 ms 

 

 

*Now re-attempt the backup, if this issue continues to happen, contact EMC Support and ask about Hot Fix 59963

 

Avamar Redirect Restores Permission Issue

I ran into a issue in the past where an Avamar user was configured at a Domain level and they were not able to perform the redirected SQL restore.  

Today I ran into the same issue and after searching some old notes, I found the fix and thought it might be useful to add it to the blog site:

The fix is to backup and then modify the following file:/usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

(There is a section which determines who can do redirected restores):

<entry key=”admin_can_direct_restores” value=”false” />
<entry key=”restore_admin_can_direct_restores”
value=”false” />

Just set the values to true and restart the MCS service and you will be good to go.

Of course doing this will clear out your Activity Log in Avamar Administrator so be warned.

Later!!!

Avamar GEN4 Network Preparation for install

I get asked quite often during the planning phase for Avamar GEN4 Installs on what the network requirements are for the software install.  I thought I would go ahead and post up some information that I usually pass along; this is for GEN4 Hardware only.

Single Node Avamar:

The minimum requirement for networking is 1 CAT6 Copper cable for the primary backup port (GB1 eth0), this is a connection to the the Customer Network Switch. If there is a second switch available to provide redundancy, then a 2nd CAT 6 Copper cable can be run to Customer Network Switch 2 for the secondary port (GB3 eth2). Backup port GB1 (eth0) and GB3 (eth2) are bonded together in the SUSE Linux OS software running on the Avamar Single Node and are an Active/Standby configuration and does not require Link Aggregation on the Customer Switches.

Multi-Node Avamar:

The minimum requirement for networking is 1 CAT6 Copper cable for the primary backup port (GB1 eth0) for each node of the Avamvar GRID including the Utility Node, storage node and NDMP physical nodes. This is a connection to the Customer Network Switch. If there is a second Customer switch available to provide redundancy, then a 2nd CAT 6 Copper cable can be ran to the Customer Network Switch 2 for the secondary port (GB3 eth2). Backup port GB1 (eth0) and GB3 (eth2) are bonded together in the SUSE Linux OS software running on the Avamar Utility and Storage Nodes and are an Active/Standby configuration and does not require Link Aggregation on the Customer Switches. So for a 1 Utility Node and 3 Storage Node GRID, 4 CAT 6 Copper cables are required for backup, for switch redundancy, 8 cables would be required. Avamar GEN 4

 

Internal Network Specifications:

The ADS Allied Telesys Switches do not require connectivity to the Customer Network All nodes are plugged in to internal dedicated ADS switches through Gb2 (eth1) and Gb4 (eth3) as primary and secondary interfaces of bond1. The internal network is a redundant, high availability, fault-tolerant network connecting all nodes in the cluster for RAIN, rebuilding, and maintenance functions. It carries all Avamar internal operations and data management traffic.

 

Example of Single Backup Network (No Redundancy) from EMC Avamar documentation.

Example of Backup Network with High Availability EMC Avamar documentation.

 

  

iDRAC Remote Management:

Through iDRAC, Administrators have full control of the server hardware and operating system from any client system running a Web browser. Barring a power outage, a network outage, or an unusual hardware problem that affects the iDRAC itself, the iDRAC console should remain available, even if the operating system is down or unresponsive. An ADS Gen4 node can communicate with a management network either by sharing the eth0 Ethernet port (Gb1) on the node or through a dedicated port (10/100 Mb integrated network interface) *If you choose to separate the iDRAC traffic to a dedicated port, then an additional CAT6 Copper Cable will be required for each node of the Avamar GRID, Utility Node, Storage Nodes or physical NDMP nodes.

How to check the status of a hot spare rebuild on a VNX

Here is a nifty Navicli command that will allow you to check the status of a hot spare rebuild on a VNX:

Open a command prompt on your workstation where Navisphere CLI is installed

Change to the following directory: C:\Program Files\EMC\Navisphere CLI

 Run this command to check the rebuild status of Bus 2, Enclosure 2, Disk14: 

NaviSECCli.exe -address “SP IP” -user sysadmin -password “password” -scope 0 getdisk 2_2_14 -rb

Here is the output:

admin -password sysadmin -scope 0 getdisk 2_2_14 -rb
Bus 2 Enclosure 2  Disk 14
Prct Rebuilt:            8180: 100

Update: Avamar 6.1 VM Image backup fails: avvcbimage Error: Failed to connect to virtual disk; You do not have access rights to this file

I have recently learned that there is now a hotfix for the issue when Windows 2008 R2 VMware Image Level backups fails with message “avvcbimage Error: Failed to connect to virtual disk; You do not have access rights to this file”.  This is common after upgrading to Avamar version 6.1.1-87.  To correct the issue, call EMC Support and ask for hot fix 47824.