h1

Avamar 7 – Cleaning up mounted snapshots, hidden snapshot and orphaned files when receiving “createSnapshot: snapshot creation failed” errors

April 3, 2014

The following are some steps for cleaning up mounted snapshots, hidden snapshot and orphaned files when Avamar VMware Image backups fail with the following error, “createSnapshot: snapshot creation failed”.  

You can edit the properties of the Avamar Proxy appliance in vCenter and if you see more than 2 vmdk’s present then could use the following steps to clean up orphaned snapshots. Please note that proxcp.jar is a powerful tool and I would highly recommend engaging EMC Support or an EMC Partner for assistance when using it. You may also want to keep it located in the /home/admin directory.  

 

  • Putty into the Avamar Utility Node as root and download the proxcp.jar file to the /tmp directory:

               cd /tmp

                curl -O ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/proxycp.jar 

  • To see a list of commands, run the following from the Avamar Utility Node:

               java -jar proxycp.jar

  • To view VM’s with hidden Snapshot Files, run this command:

              java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Snapshot

              Output example from my lab:

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

 

  • To remove original disk from the proxy that they were still mounted to, use this command.

                  java -jar proxycp.jar –listproxydisk –cleanup

                 Output example from my lab:

VDP/Nemo Detected : false

————————————————————————————

Listing single instace of a Proxy, use –all to list all the instances

====================== grif-vcenter.whughgriffin.com ============================

——————————————————————————-

Proxy Name     Grif-Proxy                         (grif-proxy-proxy-1)               

——————————————————————————-

Agent Version  : 7.0.100-427                      

Disk Key       : 2000                             

Attached Disk  : [VMFS_LOCAL] Grif-Proxy/Grif-Proxy.vmdk

Disk Key       : 2018                             

Attached Disk  : [VMFS_LOCAL] Grif-Proxy/Grif-Proxy_1.vmdk

——————————————————————————————————————————————–

Closing Connection Pool : April 3, 2014 9:31:28 AM EDT

 

  • Next clean up Avamar only snapshots with this command if any are present

                  java -jar proxycp.jar –listsnapshots –cleanup

                  Output example from my lab:

root@grif-ave:/tmp/#: java -jar proxycp.jar –listsnapshots –cleanup

======================= Starting ProxyCP ==========================

 

Proxycp : v1.89

Date : April 3, 2014 1:29:14 PM EDT

 

COMMAND : java -jar proxycp.jar –listsnapshots –cleanup 

 

VDP/Nemo Detected : false

————————————————————————————

VM Name(Avamar)                    : Grif-DC                       

VM Name(Vmware)                    : Grif-DC                       

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_NFS] Grif-DC_1/Grif-DC.vmdk

Thin Provisioned                   : true                          

Total Size (GB)                    : 40                            

—————————————————————————————-

VM Name(Avamar)                    : Grif-Desktop                  

VM Name(Vmware)                    : Grif-Desktop                  

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop.vmdk

Thin Provisioned                   : true                          

Total Size (GB)                    : 100                           

—————————————————————————————-

VM Name(Avamar)                    : Grif-Webmail                  

VM Name(Vmware)                    : Grif-Webmail                  

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail.vmdk

Thin Provisioned                   : false                         

Total Size (GB)                    : 40                            

—————————————————————————————-

VM Name(Avamar)                    : Grif-Webmail2                 

VM Name(Vmware)                    : Grif-Webmail2                 

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

Thin Provisioned                   : false                         

Total Size (GB)                    : 40                            

Hidden Snapshot File               : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

—————————————————————————————-

VM Name(Avamar)                    : Grif-Win8                     

VM Name(Vmware)                    : Grif-Win8                     

Snapshot Name                      : No Snapshots                  

Hardware Version                   : 8                             

Disk Type                          : persistent                    

File Name                          : [VMFS_LOCAL] Grif-Win8/Grif-Win8.vmdk

Thin Provisioned                   : false                         

Total Size (GB)                    : 100                           

—————————————————————————————-

 

  • Check for orphaned snapshots causing exit code 10056

                      java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Hidden

                      Output example from my lab:

root@grif-ave:/tmp/#: java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Hidden

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk

Hidden Snapshot File               : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk

 

 

  • Using the above list you may proceed using the command below to attempt to consolidate the vm’s that are failing.  Before proceeding I would check with the VMware Administrative staff and update them on the issue, you/they may be able to address the issue with a simple Storage vMotion to a different VMFS datastore and running the backup again.  The VM snapshots may also be consolidated via vCenter using the vSphere Client.  Once again, consolidating VM snapshots could have irreversible affects on the VM.  If you are not comfortable with this step, I would urge you to open a Service Ticket with VMware Support and EMC Avamar Support.

         java -jar proxycp.jar –consolidatesnapshots –vm REPLACEWITHVMNAME

         Output example from my lab:

root@grif-ave:/tmp/#: java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail

======================= Starting ProxyCP ==========================

 

Proxycp : v1.89

Date : April 3, 2014 9:33:34 AM EDT

 

COMMAND : java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail 

 

VDP/Nemo Detected : false

————————————————————————————

It is recommended that you run ‘–listproxydisk –cleanup’ option before consolildation, Please ‘CTRL C’ to exit, waiting 10 seconds

^Croot@grif-ave:/tmp/#: java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail

======================= Starting ProxyCP ==========================

 

Proxycp : v1.89

Date : April 3, 2014 9:35:14 AM EDT

 

COMMAND : java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail 

 

VDP/Nemo Detected : false

————————————————————————————

It is recommended that you run ‘–listproxydisk –cleanup’ option before consolildation, Please ‘CTRL C’ to exit, waiting 10 seconds

VM Name(Avamar)                    : Grif-Webmail                  

VM Name(Vmware)                    : Grif-Webmail                  

Snapshot Name                      : No Snapshots                  

Hidden Snapshot File               : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk

 

Consolidating Snaphot by “Taking new Snapshot & Deleting All Snapshot” !!

Please be patient, It might take a while !!!

 

101% ################################################## /

Snapshot “Avamar-Proxycp” was created.

 

Removing All the snapshots,  Please wait !!

101% ################################################## /

Removed All Snapshots : Time taken : 1 Min 44 Sec 538 ms 

 

 

*Now re-attempt the backup, if this issue continues to happen, contact EMC Support and ask about Hot Fix 59963

 

Advertisements

2 comments

  1. Just a quick word of warning that while proxycp is an amazing and useful piece of software, it is a tool that is intended for use by the support team. Use all caution when running it.


    • Thanks, please note I added the following text, “Please note that proxcp.jar is a powerful tool and I would highly recommend engaging EMC Support or an EMC Partner for assistance when using it. You may also want to keep it located in the /home/admin directory. “



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: