VMware vSphere 4 Guided Consolidation

I recently passed the VCP 410 exam and was surprised to find several questions on Guided Consolidation, a very basic tool that I had really no experience with.  To get more acquainted with it, I decided to write a blog about it.

If you are preparing for the VCP 4 exam, and have little or no knowledge about this tool, then maybe this blog may help you a little.

Guided Consolidation is a tool that will allow you to monitor a physical computer and determine it’s potential for adding to your virtual environment. I would highly recommend it for a small to mid sized business looking for some assistance with their P2V process. It has an easy to use interface with a more simplified approach than using the full VMware Capacity Planner utility.

It is free and can be installed from the vCenter installation CD.  Once installed, it can be accessed from the vCenter Home Menu.  In order to scan a physical computer, you will need to make sure the credentials provided during installation must have administrative permissions on any remote Windows systems selected for analysis.  This user account supplied during installation is used to run the “VMware vCenter Collector Service”, a Windows service used to connect to the remote systems. 

Once the tool begins to scan the system, it will gather data and display CPU information and utilization, memory information and utilization as well as the computer name.  Sometimes it may take up to 1hr before this process begins.  Be prepared to allow 24 to 48hrs for this process to complete as Guided Consolidation builds a confidence metric level.  Once a high enough confidence level is reached, the status of the system changes to “Ready for consolidation”. 

At this point, a consolidation plan is available by selecting the analyzed computer and clicking the “Plan Consolidation” button.

This plan will include a star rating which will indentify how likely the physical computer is for virtualization and will even make a recommendation for the target host.  The rating system ranges from 1 to 5 stars, with 5 stars indicating the system is a high candidate for the proposed host.  From the Consolidation Wizard, you can change several things, the name of the VM, the host being assigned too or even remove a VM altogether from the list.

Multiple systems can be converted to VM’s using this wizard, it’s a very simple process and by the click of one button you are off to the races converting your physical systems to virtual machines.

***Keep in mind that before using Guided Consolidation that a datacenter exists and a host is added to the vCenter Server inventory.  ***

I would highly recommend looking at this pdf: http://www.vmware.com/pdf/vsphere4/r40/vsp_40_admin_guide.pdf or picking up a copy of Mastering VMware vSphere 4 by Scott Lowe for more in-depth information on Guided Consolidation.

I hope this gives you a basic understanding of how Guided Consolidation can help you plan a smooth P2V process.

Thanks for stopping by!

Finding Disk Space Hogs

Have you ever experienced a situation where your workstation or server is completely out of space?  Trying to hunt down where all that valuable storage went can be frustrating.  Not to worry, there is a free tool available that will not only search and display all folders listed in order by space used, but it will also allow you to explore each folder.  Like I said before, it’s FREE!  I have included a screenshot of an example of my very own VM of XP Pro that I work on from day to day, as you can see, Documents and Settings leads the pack.

The download is available at Jam Software, link below:

http://www.jam-software.com/freeware/index.shtml

 

 

 

How to change Avamar Node Type

When running Avamar in a lab environment, I have come across situations where I want to use a node that was part of a previous Avamar configuration and make it a spare, single node server, a data node  or possibly even an accelerator node for another Avamar Data Store.

In the past I was not sure how to change the node type. 

Fortunately, I recently came across the command to make this happen.

As root, run change_nodetype in the /usr/local/avamar/src directory.

(Verify that you are logged in as root) whoami command

# /usr/local/avamar/src/change_nodetype –[type]

–[type] = the type of node to be configured.

The accepted types include:
data,axion-e (single node server),accelerator,spare,utility

As an example, in a recent situation, an Avamar storage node was mistakenly setup as an Avamar Accelerator node.  The desired setup was a RAIN configuration with a Utility Node, 4 1TB Storage Nodes and 1 Spare Node.  In our setup, we had the Utility Node, 3 1TB Storage nodes, 1 Spare node and the 1TB Accelerator Node that was configured improperly. 

To correct the issue, I performed the following tasks on the accelerator node to make it a data node:

Opened a Putty session to the Accelerator Node and logged in as root

typed the following command:

 /usr/local/avamar/src/change_nodetype –data

Afterwards, I had to update the usersettings.cfg file using vi or nano and confirm that the server name is correct by typing the following command:

nano /usr/local/avamar/etc/usersettings.cfg (as seen below):

***If you need to change the hostname or IP address as well, use the following command as root:

/usr/sbin/netconfig –ip=ipaddress –netmask=subnet mask hostname=hostname –domain=domain.name –device=eth0 –gateway=default gateway

You also want to make sure that the following configuration files are correct:

/etc/hosts
/etc/resolv.conf

/etc/sysconfig/network

You may now reboot, once the node boots back up, log in from a Putty session and verify that the notification screen states the correct node type.

From here I was able to continue with the procedures to add an additional storage node…

Reaching Avamar Zen or as EMC likes to call it “Steady State”

Now that you have implemented your Avamar System, you may become easily restless by the rapidly upsurge in storage used during the 1st to 2nd week.  Do not worry, this is common for an Avamar System as it initially goes through the process of backing up new clients during each client’s “initialization” phase where nearly every client contains high amounts of unique data.  Once clients have been backed up at least one time, you will begin to leverage the commonality feature that Avamar has to offer. 

You will begin to notice that your Avamar Server will back up less and and less unique data as the weeks pass by and the new data written to your Avamar Server will begin to plateau.  EMC refers to this as reaching a “steady state” of capacity utilization which I refer to as Avamar Zen.  You can expect for this to occur sometime after the longest retention period of your first initial backups are met.  The goal of successfully managing your Avamar Server is to achieve this Avamar Zen (steady-state) where the rate at which new data chunks added to your server are equal to or less than the rate of data chunks removed from the server during the Garbage Collection cron job that occurs for up to 2hrs during the daily maintenance schedule.

So you may be wondering, how do I tell if I have reached Avamar Zen, well not to worry, I have included two different methods for receiving this information.

• You can issue the following command /usr/local/avamar/bin/capacity.sh to ascertain whether the system is running in steady-state.  The information reported back includes amount of new data added to your Avamar System, amount of data removed, net change and clients with the highest change rate.

Example below:

 

*As you can see, the Microsoft Exchange Server listed above has the highest change rate at 84%

 

• The other option is running the DPN Summary Report from inside the Avamar Administrator.  The report will provide information about data stored in the Avamar server as well as statistical data on the individual Avamar clients. 

As you can see, the DPN Summary Report will include statistics such as daily change rate, high change rate clients, amount of data/# of clients being protected, etc.  This information can be useful in determining abnormal client behavior as well as comparisons to previous backup strategies such as tape backup.

Come back soon and we will discuss factors affecting capacity utilization and monitoring Avamar Checkpoint Overhead.

Thanks for stopping by!!!

Solution for Avamar Windows 2008 Backup and Trend Micro

I recently posted steps on how to restore a Windows 2008 Server and System State with EMC’s Avamar.  I was also made aware of another issue where the Trend Micro anti-virus client conflicted with the VSS and System State backup with the same server running Windows 2008.  I’d like to thank Stephen Saunders for passing this work around provided by Trend Micro to me, here are the steps to to alleviate this issue:

 

Problem:	After installing the Client/Server Security Agent on Windows 2008 32-bit and 64-bit versions, you can no longer backup the Windows System State.   When you run the command "WBADMIN START SYSTEMSTATEBACKUP -backuptarget:D:", the operation fails with the message Enumeration of the files failed when identifying system state files to back up.   You stopped the Trend Micro services, but this did not resolve the issue. When you uninstalled the Security Agent, the backup works just fine.
Solution:	Just disabling all the Trend Micro services does not work because the Windows System State Backup does not parse the file path of the WFBS Agent: C:\Program Files\Trend Micro\Client Server Security Agent\..\BM. This is because the backup does not recognize the (..) characters.   Before using Windows System State Backup from the command line, please do the following:

1.	Stop the Trend Micro Unauthorized Change Prevention Service.
2.	Open the Registry Editor (regedit.exe).
	Important: Always create a backup before modifying the registry.  Incorrect registry changes may cause serious issues.  Should this occur, restore it by referring to the "Restoring the Registry" Help topic in Regedit.exe or the "Restoring a Registry Key" Help topic in Regedt32.exe.
3.	Go to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TMBMServer registry hive.
4.	Look for the registry key "HomeDir" and change its value to "C:\Program Files\Trend Micro\BM".
5.	Look for the registry key "ImagePath" and change its value to "C:\Program Files\Trend Micro\BM\TMBMSRV.exe" /service.
6.	Start the Trend Micro Unauthorized Change Prevention Service.
7.	Open a DOS command window and run this command: "wbadmin systemstatebackup"
	The backup process will now work.

Recovering a Windows 2008 Server with Avamar

Thanks to Sean Dobes for sharing this procedure with me.

• Implement the Windows 2008 System State backup using the Windows 2008 native tool " wbadmin ".
• It is the replacement for "ntbackup" in Windows 2003, but works in quite different way.
• Windows 2008 System State includes additional data (e.g. Program Files), what makes it very big. 8-15GB of size in my case.
• Therefore you need to allocate additional space to create it, before Avamar starts de-duplication.
• "wbadmin" is not the default feature, so you need to install it "servermanagercmd -install Backup-Features –allsubfeatures"
• Windows by default does not allows you to store the System State backup in the system drive e.g. C: * However if you do not have another drive you can bypass it. You need to modify the Windows registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wbengine\SystemStateBackup\

Name: AllowSSBToAnyVolume

Data type: DWORD

Value data: 1

- Now you can prepare the Avamar pre-script which will create the System State backup, e.g.:

@echo off

if exist "C:\WindowsImageBackup" rmdir /S /Q C:\WindowsImageBackup"

wbadmin start systemstatebackup -quiet -backuptarget:C:

if not %errorlevel% == 0 (

echo **** System State Backup Failed ****) else (

echo **** System State Backup OK ****)

Recovery Steps – "worked in my case"

• Install Windows 2008 with the same SP as the original
• Define the same IP and hostname for the new system
• Install Avamar client and re-activate it
• Restore System State backup from Avamar -> C:\WindowsImageBackup
• Boot system in the Directory Service Restore mode
• Restore the System State:

wbadmin get versions -backuptarget:C:

wbadmin start systemstaterecovery -version:XXX -backuptarget:C:

• - Reboot the system
• - Restore the system drive e.g. C:\ with the following options:

Overwrite existing files = If modified

Restore open files = Defer until reboot

Restore hidden folders and files = Yes

• - Reboot the system and restore another data volumes (if needed)

Backing up SharePoint 2007 with Avamar

I recently had an issue at a client site using the SharePoint 2007 agent to back up an entire MOSS Farm and we continuously experienced either SQL errors or the job would just hang up.  After doing some testing, we found that several modifications were needed to be made.  The services for SQL Server service, Windows SharePoint Services Timer and the Avamar Backup Agent all need to be configured to Log In with the same AD account. 

This particular server was also using SQL 2005 Express, so we had to enable TCP/IP and Named Pipes as seen below:

You can also make the same changes for SQL Native Client Configuration.

Also, do not forget to specify a temp folder in the data set options screen for a location to store the temporary backup files (as seen below), do make sure that there is adequate available space or the backup will fail.

 

After making all of these modifications, we were able to complete a successful job in no time.

How to Modify the SP IP Addresses for an EMC Celerra (NS Proxy ARP Implementation)

I was recently configuring an EMC Celerra NS-120FC and using the Celerra Startup Assistant, I successfully configured the network settings only to notice after the fact that I had goofed on the IP addresses for the Storage Processors on the backend Clariion.  After doing some research on EMC Powerlink, I found the following steps that allowed me to successfully change the SP IP addresses.  It was a simple process that took about 1hr to complete.  Thought I’d pass along the procedures, taken from the EMC Powerlink Articles in case someone else finds themselves in the same boat as me.

How to Modify the SP IP Addresses for Celerra NS Proxy ARP Implementations

Caution: Changing SP IP Addresses in a Proxy ARP implementation requires rebooting the backend storage array SPs to change their network configuration. Only one SP will be rebooted at a time to make the process as non-disruptive as possible. While the SPs are rebooting, the backend storage array and Celerra system performance will be degraded. The estimated time clfor the upgrade is 15 minutes.

• Download the latest version of clariion_mgmt script: http://powerlink.emc.com/km/live1/en_US/Offering_Technical/Technical_Documentation/clariion_mgmt.zip
  Note:  If the system is running NAS 5.5.31, you do not have to download the clariion_mgmt.zip file. Just use the /nasmcd/sbin/clariion_mgmt executable.

 

• Unzip and copy the script to portable media (floppy diskette or CD-ROM). The script must be run on the Control Station.

 

• Insert the portable media into the Control Station.

 

• From Celerra Manager, select Celerras > <Celerra_name>> Tools > SSH Shell.

 

•  From the SSH session, log in to the primary Control Station as nasadmin. Change to the root user by running the following command:
  su root

 

• Mount the media on the Control Station (/sbin/mount /mnt/floppy or /sbin/mount /mnt/cdrom).

 

• Change the working directory to the mount location (cd /mnt/floppy or cd /mnt/cdrom).

 

• Copy the ‘new’ clariion_mgmt script to the following directories, overwriting the existing version: /nasmcd/sbin and /nas/sbin
  Note:  You must use only the Powerlink version of the clariion_mgmt script when converting a 5.5.30 system from NAT to Proxy ARP

 

• Confirm the current implementation of the system:

  # /nasmcd/sbin/clariion_mgmt -info
  Public IP address for SPA: 10.64.247.147
  Public IP address for SPB: 10.64.247.148
  Start on boot            : yes
  Current implementation   : Proxy-ARP
  Status                   : Started

 

• Stop Clariion MGMT

# /nasmcd/sbin/clariion_mgmt -stop
Checking if running as root…yes
Checking if model is supported…yes
Checking for integrated system…yes
Checking if interface eth3 is configured…yes
Checking if SP (10.64.247.147) is up…yes
Checking if SP (10.64.247.148) is up…yes
Changing SPA IP from 10.64.247.147 to 192.168.1.200 (subnetmask 255.255.255.0, gateway 192.168.1.104)
Waiting for SPA to go down…done
Waiting for SPA to come back up……………done
Removing host specific route for SPA
Removing rules that allow outbound traffic from SPA
Removing ARP entry for SPA
Updating /etc/hosts entry for SPA
Changing SPB IP from 10.64.247.148 to 192.168.2.201 (subnetmask 255.255.255.0, gateway 192.168.2.104)
Waiting for SPB to go down…done
Waiting for SPB to come back up……………done
Removing host specific route for SPB
Removing rules that allow outbound traffic from SPB
Removing ARP entry for SPB
Updating /etc/hosts entry for SPB
Updating NAS database with new CLARiiON IP addresses
Updating NAS cache of CLARiiON password

(You may not have to complete the next step).

• Enter the Global CLARiiON account information
  Username: nasadmin
  Password: ********              Retype your response to validate
  Password: ********
  Setting security information for CK200073500354
  done
  id       = 1
  serial_number        = CK200073500354
  name     = CK200073500354
  acl      = 0
  done

 

• Verify that Proxy ARP is no longer configured:

  # /nasmcd/sbin/clariion_mgmt -info
  Error 12: Not configured
  done

 

• Reconfigure the system for Proxy ARP using the new SP IP addresses:

  # /nasmcd/sbin/clariion_mgmt -start -spa_ip <new IP> -spb_ip <new IP> -use_proxy_arp
  Checking if running as root…yes
  Checking if model is supported…yes
  Checking for integrated system…yes
  Checking if interface eth3 is configured…yes
  Checking if interface eth3:1 is configured…no
  Checking if interface eth3:2 is configured…no
  Checking if SP (192.168.1.200) is up…yes
  Checking if SP (192.168.2.201) is up…yes
  Checking if a gateway is setup for eth3…yes
  Adding host specific route for SPA
  Adding rules to allow outbound traffic from SPA
  Adding ARP entry for SPA
  Updating /etc/hosts entry for SPA
  Changing SPA IP from 192.168.1.200 to 10.64.247.141 (subnet mask 255.255.255.0, gateway 10.64.247.1)
  Waiting for SPA to go down…done
  Waiting for SPA to come back up……………..done
  Adding host specific route for SPB
  Adding rules to allow outbound traffic from SPB
  Adding ARP entry for SPB
  Updating /etc/hosts entry for SPB
  Changing SPB IP from 192.168.2.201 to 10.64.247.142 (subnet mask 255.255.255.0, gateway 10.64.247.1)
  Waiting for SPB to go down…done
  Waiting for SPB to come back up……………..done
  Updating NAS database with new CLARiiON IP addresses
  Updating NAS cache of CLARiiON password

 

 

*** (The next step may not be required). 

• Enter the Global CLARiiON account information
  Username: nasadmin
  Password: ********              Retype your response to validate
  Password: ********
  Setting security information for CK200073500354
  done
  id       = 1
  serial_number        = CK200073500354
  name     = CK200073500354
  acl      = 0
  done
  Discovering storage (may take several minutes)
  Error 5017: storage health check failed
  CK200073500354  write cache disabled
  /nas/bin/nas_storage -check CK200073500354 failed! You will have to run this manually!

 

• Re-check for the new configured SP IP Addresses

  # /nasmcd/sbin/clariion_mgmt -info
  Public IP address for SPA: 10.64.247.141
  Public IP address for SPB: 10.64.247.142
  Start on boot            : yes
  Current implementation   : Proxy-ARP
  Status                   : Started

 

 

• Verify the addresses on the Celerra side in /etc/hosts and update if necessary.

  [nasadmin@cel9cs0 nasadmin]$ cd /etc
  [nasadmin@cel9cs0 etc]$ su
  Password:
  [root@cel9cs0 etc]# cp hosts hosts.old
  [root@cel9cs0 etc]# ls
  .
  .
  .
  hosts                 motd                   sudoers
  hosts.allow           mtab                   sysconfig
  hosts.deny            mtools.conf            sysctl.conf
  hosts.old
  [root@cel9cs0 etc]# vi hosts
  # Do not remove the following line, or various programs
  # that require network functionality will fail.
  127.0.0.1               localhost.localdomain localhost
  10.64.99.90             cel9cs0 cel9cs0.nas2003.com
  10.64.99.148    A_APM00096100732        SPA     # CLARiiON SP                   <—– Change IP address here.
  10.64.99.149    B_APM00096100732        SPB     # CLARiiON SP                    <—– Change IP address here.
  # Internal DART Server Primary Network
  192.168.1.1  server_1   #DART_data_mover_1
  192.168.1.2  server_2   #DART_data_mover_2
  192.168.1.3  server_3   #DART_data_mover_3

   

• Do a re-discover from the Celerra to the CLARiiON so as to re-discover the backend using these new IP addresses:

  [root@cel9cs0 etc]# nas_storage -check -all
  
  Discovering storage (may take several minutes)

  done

Avamar Server Configurations to keep in mind!

When building out your Avamar Server, it’s important to keep in mind the flavors of an Avamar Server.  You have the following 2 types of configurations:

Non-RAIN configurations consist of either a single stand-alone node or one utility node and two data storage nodes (1×2 Non-RAIN). In single node configurations, both utility and data functions are provided in the single node.  The 1×2 Non-RAIN Avamar Server works much like RAID0 with disks, backup data is evenly distributed between both storage nodes, so if there is a node failure, your Avamar Server has now been compromised.  This is the reason why EMC requires Replication as a form of fault tolerance in a Non-RAIN Avamar Server.  By replicating your Non-RAIN Avamar Server to an offsite location, you will be able to recover from a failed node at the source by replacing the node then replicating back from the offsite location.

****There is no need for a 1×2 Avamar Server with a spare node, in this case, if there is a node failure, you have lost your backup data because Avamar stores the backed up data in stripes distributed evenly between all storage nodes.  A 2 storage node configuration is similar in functionality to RAID0, data is striped across the 2 Avamar nodes but there is no parity in place.  Keep this in mind when planning out the Avamar Server that best fits your environment and be sure to use replication if a Non-RAIN Avamar Server is being used to backup your production site.

RAIN configurations include one utility node, a spare node, 3 or more data storage nodes, plus a spare data node. Currently, the largest standard configuration consists of 16 data nodes, 1 utility node, and 1 spare data node. In a multi-node system, the nodes operate together as one server. The hostname and IP address of the utility node is the identity of the Avamar server for access and client/server communication. Avamar load balances data across all available nodes in a server. With node architecture, Avamar can be easily scaled by adding more nodes.  In a situation where you experience a node failure, the Avamar Server will continue to function in a degraded mode until the node is replaced with the spare node.

Here is a graphical representation of the types of Avamar Servers discussed above:

Using an NDMP Accelerator with your Avamar

In an Avamar System, the NDMP Accelerator is used for backup of EMC Celerra IP storage systems and Network Appliance filers.

The NDMP accelerator is a dedicated single node that provides a complete backup and recovery solution. The NDMP accelerator acts as a pass-through conduit from the NAS device to the Avamar server and can support multiple NAS storage devices.

Here are the steps I followed to install a NDMP Accelerator for a client with an EMC Celerra.

 

Enabled NDMP services on the EMC Celerra and rebooted the data mover

Setup ndmp user on the Celerra

• Open a Putty Session to the Celerra/NetApp

• On Celerra type:

su

/nas/sbin/server_user server_2 –add –md5 –password ndmp

Set User ID= 1000

Set Group ID= 1000

Home Directory: (leave blank)

Set ndmp password (must be 6 to 8 characters long)

Open Putty session to NDMP Accelerator

Change location to /usr/tmp

Open WinScp session with NDMP Accelerator

Copy Linux 4 64bit clients for NDMP and Linux to the /usr/tmp folder on the NDMP Accelerator node

Switch back to Putty session with NDMP Accelerator

Make sure path is /usr/tmp

Install rpm files for Linux client and NDMP client, type:

rpm –ivh <linuxclientname.rpm>

rpm –ivh <NDMPLinuxclientname.rpm

Run NDMP installation, type:

avsetupndmp

The Avamar NDMP Accelerator will be configured using this script. 

First you will need to stop all running agents on the accelerator, a prompt will then ask for the Avamar server’s (utility node) DNS name or IP address. 

You will then be prompted for the Avamar server’s administrator password.

NOTE: this is not the OS root password, the Avamar’s default password is 8RttoTriz

Next, you can enable multiple simultaneous backups, keep in mind this functionality requires at least 8GB of RAM.  This shouldn’t be a problem since most accelerators come with 32GB of RAM.

You will now be presented with a menu with the following choices:

Add a new system

(choices are: EMC Celerra, NetAPP Filer or Novell Netware),

Edit an existing system,

Remove a system from the list

Exit

When adding a new EMC Celerra system, you will need  the NDMP filer IP address or hostname (Note: this is not the IP address of the control station, but the IP address of an interface on the data mover).

You will then be prompted for the ndmp user password that you assigned in the beginning steps.  Re-enter the password to verify you typed the password correct. 

You will then need to specify the correct password encoding scheme, the only options are “md5” or “text”, we used “md5”above.

You will then need to enter a routable DNS name to be used for the filer.

Wait for avsetupndmp to complete.  Add any additional Filers then choose 4 to Exit.

The last step is to register and activate the accelerator with the Avamar Server, this is performed by running a script using the following command from the accelerator node’s putty session:

avregister

Provide the Administrator Server’s host name (Avamar Utility Node)  and assign the domain you would like to assign the accelerator node client too.  The default is “clients”, if you choose to change the default to a domain that was created under the root domain, omit the / and just specify the “root name”, including “/” will cause the register process to fail.

Press “Enter” and you should see results similar to this:

avagent.d Info: Server stopped. [ OK ]
avagent Info <5241>: Logging to /usr/local/avamar/var/avagent.log
avagent.d Info: Client activated successfully. [ OK ]
avagent Info <5241>: Logging to /usr/local/avamar/var/avagent.log
avagent Info <5417>: daemonized as process id 3385
avagent.d Info: Server started. [ OK ]
Registration Complete.

 

You are now ready to backup your EMC Celerra!