The following are some steps for cleaning up mounted snapshots, hidden snapshot and orphaned files when Avamar VMware Image backups fail with the following error, “createSnapshot: snapshot creation failed”.
You can edit the properties of the Avamar Proxy appliance in vCenter and if you see more than 2 vmdk’s present then could use the following steps to clean up orphaned snapshots. Please note that proxcp.jar is a powerful tool and I would highly recommend engaging EMC Support or an EMC Partner for assistance when using it. You may also want to keep it located in the /home/admin directory.
- Putty into the Avamar Utility Node as root and download the proxcp.jar file to the /tmp directory:
cd /tmp
curl -O ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/proxycp.jar
- To see a list of commands, run the following from the Avamar Utility Node:
java -jar proxycp.jar
- To view VM’s with hidden Snapshot Files, run this command:
java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Snapshot
Output example from my lab:
Hidden Snapshot File : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop-000002.vmdk
Hidden Snapshot File : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk
Hidden Snapshot File : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk
- To remove original disk from the proxy that they were still mounted to, use this command.
java -jar proxycp.jar –listproxydisk –cleanup
Output example from my lab:
VDP/Nemo Detected : false
————————————————————————————
Listing single instace of a Proxy, use –all to list all the instances
====================== grif-vcenter.whughgriffin.com ============================
——————————————————————————-
Proxy Name Grif-Proxy (grif-proxy-proxy-1)
——————————————————————————-
Agent Version : 7.0.100-427
Disk Key : 2000
Attached Disk : [VMFS_LOCAL] Grif-Proxy/Grif-Proxy.vmdk
Disk Key : 2018
Attached Disk : [VMFS_LOCAL] Grif-Proxy/Grif-Proxy_1.vmdk
——————————————————————————————————————————————–
Closing Connection Pool : April 3, 2014 9:31:28 AM EDT
- Next clean up Avamar only snapshots with this command if any are present
java -jar proxycp.jar –listsnapshots –cleanup
Output example from my lab:
root@grif-ave:/tmp/#: java -jar proxycp.jar –listsnapshots –cleanup
======================= Starting ProxyCP ==========================
Proxycp : v1.89
Date : April 3, 2014 1:29:14 PM EDT
COMMAND : java -jar proxycp.jar –listsnapshots –cleanup
VDP/Nemo Detected : false
————————————————————————————
VM Name(Avamar) : Grif-DC
VM Name(Vmware) : Grif-DC
Snapshot Name : No Snapshots
Hardware Version : 8
Disk Type : persistent
File Name : [VMFS_NFS] Grif-DC_1/Grif-DC.vmdk
Thin Provisioned : true
Total Size (GB) : 40
—————————————————————————————-
VM Name(Avamar) : Grif-Desktop
VM Name(Vmware) : Grif-Desktop
Snapshot Name : No Snapshots
Hardware Version : 8
Disk Type : persistent
File Name : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop.vmdk
Thin Provisioned : true
Total Size (GB) : 100
—————————————————————————————-
VM Name(Avamar) : Grif-Webmail
VM Name(Vmware) : Grif-Webmail
Snapshot Name : No Snapshots
Hardware Version : 8
Disk Type : persistent
File Name : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail.vmdk
Thin Provisioned : false
Total Size (GB) : 40
—————————————————————————————-
VM Name(Avamar) : Grif-Webmail2
VM Name(Vmware) : Grif-Webmail2
Snapshot Name : No Snapshots
Hardware Version : 8
Disk Type : persistent
File Name : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk
Thin Provisioned : false
Total Size (GB) : 40
Hidden Snapshot File : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk
—————————————————————————————-
VM Name(Avamar) : Grif-Win8
VM Name(Vmware) : Grif-Win8
Snapshot Name : No Snapshots
Hardware Version : 8
Disk Type : persistent
File Name : [VMFS_LOCAL] Grif-Win8/Grif-Win8.vmdk
Thin Provisioned : false
Total Size (GB) : 100
—————————————————————————————-
- Check for orphaned snapshots causing exit code 10056
java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Hidden
Output example from my lab:
root@grif-ave:/tmp/#: java -jar proxycp.jar –listsnapshots 2>/dev/null | grep Hidden
Hidden Snapshot File : [VMFS_LOCAL] Grif-Desktop/Grif-Desktop-000002.vmdk
Hidden Snapshot File : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk
Hidden Snapshot File : [VMFS_LOCAL] Grif-DC01/Grif-DC01-000001.vmdk
- Using the above list you may proceed using the command below to attempt to consolidate the vm’s that are failing. Before proceeding I would check with the VMware Administrative staff and update them on the issue, you/they may be able to address the issue with a simple Storage vMotion to a different VMFS datastore and running the backup again. The VM snapshots may also be consolidated via vCenter using the vSphere Client. Once again, consolidating VM snapshots could have irreversible affects on the VM. If you are not comfortable with this step, I would urge you to open a Service Ticket with VMware Support and EMC Avamar Support.
java -jar proxycp.jar –consolidatesnapshots –vm REPLACEWITHVMNAME
Output example from my lab:
root@grif-ave:/tmp/#: java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail
======================= Starting ProxyCP ==========================
Proxycp : v1.89
Date : April 3, 2014 9:33:34 AM EDT
COMMAND : java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail
VDP/Nemo Detected : false
————————————————————————————
It is recommended that you run ‘–listproxydisk –cleanup’ option before consolildation, Please ‘CTRL C’ to exit, waiting 10 seconds
^Croot@grif-ave:/tmp/#: java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail
======================= Starting ProxyCP ==========================
Proxycp : v1.89
Date : April 3, 2014 9:35:14 AM EDT
COMMAND : java -jar proxycp.jar –consolidatesnapshots –vm Grif-Webmail
VDP/Nemo Detected : false
————————————————————————————
It is recommended that you run ‘–listproxydisk –cleanup’ option before consolildation, Please ‘CTRL C’ to exit, waiting 10 seconds
VM Name(Avamar) : Grif-Webmail
VM Name(Vmware) : Grif-Webmail
Snapshot Name : No Snapshots
Hidden Snapshot File : [VMFS_LOCAL] Grif-Webmail/Grif-Webmail-000002.vmdk
Consolidating Snaphot by “Taking new Snapshot & Deleting All Snapshot” !!
Please be patient, It might take a while !!!
101% ################################################## /
Snapshot “Avamar-Proxycp” was created.
Removing All the snapshots, Please wait !!
101% ################################################## /
Removed All Snapshots : Time taken : 1 Min 44 Sec 538 ms
*Now re-attempt the backup, if this issue continues to happen, contact EMC Support and ask about Hot Fix 59963