h1

Avamar Server Configurations to keep in mind!

June 24, 2009

When building out your Avamar Server, it’s important to keep in mind the flavors of an Avamar Server.  You have the following 2 types of configurations:

Non-RAIN configurations consist of either a single stand-alone node or one utility node and two data storage nodes (1×2 Non-RAIN). In single node configurations, both utility and data functions are provided in the single node.  The 1×2 Non-RAIN Avamar Server works much like RAID0 with disks, backup data is evenly distributed between both storage nodes, so if there is a node failure, your Avamar Server has now been compromised.  This is the reason why EMC requires Replication as a form of fault tolerance in a Non-RAIN Avamar Server.  By replicating your Non-RAIN Avamar Server to an offsite location, you will be able to recover from a failed node at the source by replacing the node then replicating back from the offsite location.

****There is no need for a 1×2 Avamar Server with a spare node, in this case, if there is a node failure, you have lost your backup data because Avamar stores the backed up data in stripes distributed evenly between all storage nodes.  A 2 storage node configuration is similar in functionality to RAID0, data is striped across the 2 Avamar nodes but there is no parity in place.  Keep this in mind when planning out the Avamar Server that best fits your environment and be sure to use replication if a Non-RAIN Avamar Server is being used to backup your production site.

RAIN configurations include one utility node, a spare node, 3 or more data storage nodes, plus a spare data node. Currently, the largest standard configuration consists of 16 data nodes, 1 utility node, and 1 spare data node. In a multi-node system, the nodes operate together as one server. The hostname and IP address of the utility node is the identity of the Avamar server for access and client/server communication. Avamar load balances data across all available nodes in a server. With node architecture, Avamar can be easily scaled by adding more nodes.  In a situation where you experience a node failure, the Avamar Server will continue to function in a degraded mode until the node is replaced with the spare node.

Here is a graphical representation of the types of Avamar Servers discussed above:

image

h1

Using an NDMP Accelerator with your Avamar

June 9, 2009

In an Avamar System, the NDMP Accelerator is used for backup of EMC Celerra IP storage systems and Network Appliance filers.

The NDMP accelerator is a dedicated single node that provides a complete backup and recovery solution. The NDMP accelerator acts as a pass-through conduit from the NAS device to the Avamar server and can support multiple NAS storage devices.

Here are the steps I followed to install a NDMP Accelerator for a client with an EMC Celerra.

 

Enabled NDMP services on the EMC Celerra and rebooted the data mover

Setup ndmp user on the Celerra

  • Open a Putty Session to the Celerra/NetApp
  • On Celerra type:

su

/nas/sbin/server_user server_2 –add –md5 –password ndmp

Set User ID= 1000

Set Group ID= 1000

Home Directory: (leave blank)

Set ndmp password (must be 6 to 8 characters long)

Open Putty session to NDMP Accelerator

Change location to /usr/tmp

Open WinScp session with NDMP Accelerator

Copy Linux 4 64bit clients for NDMP and Linux to the /usr/tmp folder on the NDMP Accelerator node

Switch back to Putty session with NDMP Accelerator

Make sure path is /usr/tmp

Install rpm files for Linux client and NDMP client, type:

rpm –ivh <linuxclientname.rpm>

rpm –ivh <NDMPLinuxclientname.rpm

Run NDMP installation, type:

avsetupndmp

The Avamar NDMP Accelerator will be configured using this script. 

First you will need to stop all running agents on the accelerator, a prompt will then ask for the Avamar server’s (utility node) DNS name or IP address. 

You will then be prompted for the Avamar server’s administrator password.

NOTE: this is not the OS root password, the Avamar’s default password is 8RttoTriz

Next, you can enable multiple simultaneous backups, keep in mind this functionality requires at least 8GB of RAM.  This shouldn’t be a problem since most accelerators come with 32GB of RAM.

You will now be presented with a menu with the following choices:

Add a new system

(choices are: EMC Celerra, NetAPP Filer or Novell Netware),

Edit an existing system,

Remove a system from the list

Exit

When adding a new EMC Celerra system, you will need  the NDMP filer IP address or hostname (Note: this is not the IP address of the control station, but the IP address of an interface on the data mover).

You will then be prompted for the ndmp user password that you assigned in the beginning steps.  Re-enter the password to verify you typed the password correct. 

You will then need to specify the correct password encoding scheme, the only options are “md5” or “text”, we used “md5”above.

You will then need to enter a routable DNS name to be used for the filer.

Wait for avsetupndmp to complete.  Add any additional Filers then choose 4 to Exit.

The last step is to register and activate the accelerator with the Avamar Server, this is performed by running a script using the following command from the accelerator node’s putty session:

avregister

Provide the Administrator Server’s host name (Avamar Utility Node)  and assign the domain you would like to assign the accelerator node client too.  The default is “clients”, if you choose to change the default to a domain that was created under the root domain, omit the / and just specify the “root name”, including “/” will cause the register process to fail.

Press “Enter” and you should see results similar to this:

avagent.d Info: Server stopped. [ OK ]
avagent Info <5241>: Logging to /usr/local/avamar/var/avagent.log
avagent.d Info: Client activated successfully. [ OK ]
avagent Info <5241>: Logging to /usr/local/avamar/var/avagent.log
avagent Info <5417>: daemonized as process id 3385
avagent.d Info: Server started. [ OK ]
Registration Complete.

 

You are now ready to backup your EMC Celerra!

h1

Avamar System Overview

June 3, 2009

Avamar Overview

Here is a overview I put together on an Avamar System, some of the functions held by the different types of nodes and examples of the 2 different types of nodes and backups.

Avamar Node Types and Processes

There are several different types of Nodes within an Avamar Server.  The Utility node is the identity of the Avamar Server and provides the bulk of the internal Avamar server processes such as: Avamar Administrator (used to manage the Avamar Server from a gui), cron jobs, DNS, NTP, external authentication, web access, MCS and EMS.

MCS (Management Console Server) provides centralized management including scheduling of backups, restore of backups, monitoring and reporting.

EMS (Enterprise Manager Server) provides web based management for multiple Avamar Servers as well as monitoring and configuration for Avamar Replication.

Web Access provides access to documentation, backup plug-ins as well as remote File System  restore access to end users.

Are next type of node is the Storage Node, which run a process called gsan.  This service communicates with the avtar command on the individual backup clients.  Avtar communicates with a storage nodes gsan process, then that storage node spreads the data across the available data nodes.

A Spare Node is an active node that  is present in a multi-node RAIN (Redundant Array of Independent Nodes) grid.  The interesting thing about the Spare Node is that it is NOT a hot spare.  There is a procedure in place that needs to be followed when a failed Storage Node occurs.  In fact, the Avamar Server doesn’t even care if the Spare Node is powered on, so if you are a “Green” conscience company, feel free to leave the Spare Node powered down.  EMC Best Practice is to leave the Spare Node up and active though.

Avamar Backups clients can be installed with 2 different types of client plug-ins: File System and Databases.  The Avamar client uses 2 processes in the backup and restore process: Avtar and Avagent.  The Avagent process listens for backup/restore work orders from the MCS service on the Utility Node using port 28002 and executes the avtar command that handles the backup/restore processes and communicates with the gsan service on the storage nodes.  This process is depicted in the included picture above.

Lastly, in a previous blog, I discussed the 2 different licensable options for Avamar and how the disks are comprised in each Storage Nodes type.  These are depicted in the picture above as well as a breakdown of how the disks are structured. 

Please stop back by later, I plan to talk about the NDMP Accelerator and it’s role in the Avamar System.   

h1

Avamar Tidbits

May 18, 2009

I recently have been working with Avamar.  If you are not familiar with Avamar, it’s EMC’s client-server backup and restore solution that includes a unique global data de-duplication technology.  What makes Avamar so unique is that it identifies redundant data at the source, not only reducing the amount of backup data traveling across your network, but drastically decreasing the amount of time for backups to complete. 

To learn more of the basics of Avamar, see the following blog’s:

http://jasonnash.wordpress.com/2009/03/16/avamar-and-vmware/

http://mitchellzblog.blogspot.com/2008/12/what-exactly-is-avamar.html

So, as I was saying, I have been recently working with Avamar and I have come across some things that I have found quite interesting.  I thought I would share some of these factoids, let me know what you think!

One area of great concern when backing up to disk are the multiple points of failure that exist.  To battle these concerns, Avamar provides fault tolerance at several different levels:

  • Avamar ensures protection from disk and data corruption through the use of RAID (redundant array of independent disks).  The type of RAID depends on the particular node type.   The first thing you must know is that all storage nodes have 6 physical disks. Beginning with Avamar release 4.0, two sizes are supported: nodes with 1TB of licensable capacity or 2TB of licensable capacity. Single-node servers with a capacity of 1TB actually have higher performance disks than the 2TB flavor.  The 1TB storage nodes are 6 300GB 15k SAS drives setup in RAID-5, configured into 4 Luns (virtual disks).  If you opt for the 2TB capacity, the storage node will have 6 SATA disks configured with 3 RAID-1 Luns. Just FYI, the Utility node and NDMP accelerator nodes both only have 2 146GB physical disks configured with RAID-1.
  • In an multi-node system (1 utility node & 3 storage nodes or more), Avamar provides failover and fault tolerance across all nodes using RAIN (redundant array of independent nodes).  If a node failure occurs, the Avamar Server continues to function, during this time, backup data for recovery will be reconstructed on the remaining nodes using parity. Once the failed node is replaced (a spare node is always included in a RAIN configuration), the capacity across all disks can be rebalanced using a very simple process.  
  • Stand-alone configurations or what is referred to as 1×2 configurations (which are made up of 1 utility node and 2 storage nodes) do not have the luxury of the RAIN configuration to protect from a node failure so EMC requires that the Avamar be configured with Replication.  Replication is the 3rd type of fault tolerance and is optional for RAIN configurations but come in handy when you have multiple remote offices that you want to replicate to a centrally managed location for protection against server failure.  If your stand-alone Avamar Server or 1×2 Avamar Server were to fail, your backups would be unavailable until the failed node was replaced and the disaster recovery Avamar (that you were replicating too) was replicated back to the production site.  In summary, to take advantage of the RAIN fault tolerance, it is important that you remember that you MUST have a Utility Node and atleast 3 Storage Nodes.  As for the stand-alone Avamar Server or 1×2 Avamar Server, RAIN is not available and it is highly recommended to use replication to a second Avamar Server onsite or preferably offsite for fault tolerance.
  • Avamar protects itself from operational failures through the use of Checkpoints, or read-only snapshots of the server taken twice a day that can be used for server rollbacks.  The checkpoints are actually hard-links to all of the stripes that are validated by performing hash file system (HFS) checks.  HFS checks are ran once a day during the morning cron jobs. If there are backups running when a checkpoint begins, they will be suspended and the system becomes read-only until the checkpoint is complete then the system is made read-write and backups will resume.  Avamar retains the last 2 checkpoints and the last validated checkpoint.  Checkpoints can be created manually at anytime by Administrators as well as deleted. 

Another area that I find quite interesting is actually the key feature of Avamar, the De-duplication process.  If you are like me, you probably wonder how the de-duplication process works.    I will step you through the process that the Avamar server goes through during a backup. 

  1. During a scheduled backup or a backup on demand, the administrator server generates a work order then either pages the client or the client checks in periodically to pickup the work order.
  2. The client’s local file cache is checked to see if the files being backed up have been backed up before, in turn, skipping files that have been previously backed up.
  3. If there is no match in the local cache, the file is divided into variable or different sized chunks.
  4. Each data chunk may be compressed.  A compressed data chunk is then hashed into an atomic hash.
  5. Atomic hashes are combined to create composites.
  6. The atomic and composite hashes are compared to the entries in the client’s hash cache to determine if it has been stored before.
  7. If there is no match, the hash cache is updated and sent to the Avamar server to check if it is present.
  8. If there is no match on the Avamar server, then the hash and the data are both sent to the Avamar server.
  9. This process continues until a single root hash for the backup is created, a single root hash is actually a full point in time backup that through the series of hashes, links to all the files and data that comprise the backup.

Another topic that I have come across with the Avamar that seems to be a hot topic are the differences between backing up databases as opposed to file systems. There are more unique data located inside databases than in file systems.  EMC estimates that the daily change rate is typically around 3% versus 0.3% for file systems.   The first backup or initial backup usually sees a initial change rate of 65% for databases as opposed to 35% for file systems.  EMC also estimates that 100GB/hour is typical for database backups.  It is a best practice to limit Avamar to databases 500GB in size or less due to the fact that backup windows are usually around 10 hours.  Keep in mind, if you are using replication, only backups that are complete are replicated, unfinished backups are replicated the next day.   Avamar will still find and eliminate redundant data speeding up backup speeds and reducing overall storage requirements by tenfold; the drag is waiting for the entire databases to be read.  A helpful thing to consider, offload backup processing to another machine may reduce CPU utilization on the database server performing the backup.

One last interesting tidbit for the today’s blog is the subject of when to schedule your backups.  We discussed earlier that when the checkpoints and HFS check occurs, backups are paused, but backups cannot be run during garbage collection, in fact, garbage collection will not start if backups are running.  Garbage collection is the process the Avamar goes though to remove expired and unused chunks.  The daily maintenance schedule that includes the garbage collection process starts at 6am.  The morning maintenance flows as follows: garbage collection runs for 2hrs then stops no matter whether all expired data is remove or not, it is then followed by a checkpoint, HFS check and a second checkpoint. 

To ensure backups and daily maintenance do not affect each other, I would suggest to start backups after 8pm for a duration for no more than 10hrs; be sure to configure backups to stop by 6am.  After the first initial backup is performed for each client, this should not be an issue because most backups will complete in the first 2 to 3hrs.  If replication is being used, overlapping this with backup activities is unlikely to affect the amount of time required to perform replications and will only have a slight impact on backup performance.  For the record, 18 simultaneous backups can be performed per storage node, unless a checkpoint or HFS check occur.     

Well that is it for today, be sure to check back from time to time for updated interesting facts about Avamar.

h1

Windows 7 Caveat no more!

April 8, 2009

After installing Windows 7 Beta inside VMware Workstation 6.5, I became extremely comfortable with it, in fact, I was even bold enough to install it on my pc at home a couple of months ago. 

After a couple of days of performing my day to day activities, I finally found a small caveat, I could no longer use Magic ISO to mount ISO files.  Now, I was rather impressed to find that Windows 7  provides the capability to burn ISO files using the Windows Disc Image Burner, however, I was a little annoyed that Microsoft still had no solution to mount the ISO files.  See the following site for a quick ” How To…”. http://www.sevenforums.com/tutorials/548-burn-disc-image-iso-img-file.html

After doing some searching online, I could never come across a solution to mount ISO’s so I eventually restored Windows Vista back onto my pc.

 Well today I decided to check in and see if Magic ISO had ever released an update and low and behold there is now a download that supports Windows 7 as seen below.

iso

 

The virtual CD/DVD-ROM application is free and can be downloaded at:

http://www.magiciso.com/tutorials/miso-magicdisc-overview.htm?=magiciso

h1

Part Two: Need a free software SAN Solution for VMware ESX Server?

April 3, 2009

 

Now that we have a working SAN for our VMware Environment, let’s continue with the bulk of the work which is configuring Openfiler to work with VMware ESX Server.

When we left off the other day, Openfiler VM should have been powered on and the the management console screen should look similar to this:

***Make sure DHCP is enabled in your environment

open filer console

One thing you will notice is that is that a dynamic IP address was obtained, this can be adjusted from the Web administration GUI and I will touch more on this later.

So let’s go ahead and open a web browser and connect to the Web administration GUI.

gui login

The default Username is openfiler and the Password is password.

Once you are logged in, you will arrive at the Status page.

The first thing we can do is adjust the clock settings.  To do this, choose the System tab up top and you will see in the picture below an example of the right side of the screen underneath the System section where there is a link to to Clock Setup.

clock setup I

 

After browsing to the Clock Setup page, you can see that the system clock and time zone can be set manually or synchronized with a NTP timeserver (a few NTP servers are even listed as examples).

clock setup 3

Now we can move along and configure the Local Area Network settings.  to do this, simply browse to the Network Setup link underneath the System section to the right.

You will see that there are several sections that need to be modified:

netsetup1

First you will want to assign a Hostname, DNS servers for name resolution and a Default Gateway (as seen below).

***Be sure to add an A Record onto your DNS Server.

netsetup2

Next you will want to configure a static IP address as opposed to the dynamic IP address obtained earlier.  Underneath the Network Interface Configuration section you will see your current interface settings and you will notice that the Boot Protocol is set to DHCP, let’s go ahead and change that by clicking on Configure underneath the Edit column:

netsetup4

You will need to select Static from inside the following drop down box:

netsetup5

Assign your static IP address, Netmask and MTU settings and click on confirm.

netsetup6

You will return to the Network Settings page and you will notice that your static interface has now been created.

The last section, Network Access Configuration can be used add additional network interfaces, by choosing a Name for the interface, set your Network/Host and Netmask and choose Share as seen below:netsetup8

Okay, so now we should have Network Interface up and running, you should now be able to ping the static IP address you assigned:

ping

We can now continue on and configure our Volumes, If you browse to the Volumes tab up top, you will arrive to the Volumes Section.

volume1

the first thing we will want to do is create a physical volume, to do this, open up the virtual machine settings from inside VMware Workstation for your Openfiler virtual appliance.

You will notice you currently only have 1 virtual disk.  Let’s click on Add as seen below:

harddrive1 

This will bring up the Add Hardware Wizard, choose Hard Disk and click Next:

harddrive2

This will bring up the Select a Disk Wizard, you can choose Create a new virtual disk, Use an existing disk or Use a physical disk.  We are going to Create a new virtual disk and click Next.

harddrive3

The Select a Disk Type window will now appear, leave the Virtual disk type set to SCSI, additionally, you can enable Independent if you do not want your physical SAN storage to be affected by snapshots.  Click Next.

harddrive4

The Specify Disk Capacity window will now open, set your Disk size, notice the option for Allocate all disk space now, this is important, if you want to assign a maximum capacity, but do not want to allocated all of the disk space immediately, leave this option unchecked.  This is great if you are only planning on using Openfiler in a testing or lab environment.  Choose Next.

harddrive5

The Specify Disk File window will now appear, you can either leave the vmdk file located locally with the virtual machine files or choose a new location to store the vmdk file.  In my case, I chose a new location on a different disk where I had more available free space to accommodate 100Gb.  Once this is set, click Finish.

harddrive7

Now, let’s return to the Openfiler Web administration GUI and return to the Network section where he left off.  From here, let’s choose create new physical volumes from the Create New Volume Group section.

volume1

This will bring up the Block Device Management section and you will now see a list of your physical drives.  We are going to create a partition on the newly created virtual disk we created in the previous step, so underneath the Edit Disk column, choose the virtual disk you just recently created.

volume2

The Create a partition window will not appear, choose the proper settings for your environment and choose Create.

partition

If you return to the Volumes Section up top, you will see the Create a new volume group section.  We now want to create a volume group, assign a Volume group name, check the physical volume and click Add volume group as seen below.

volume3

The newly created Volume Group will now appear in the Volume Group Management section up top.

volume4

Return to the Volumes Section above once more and the newly created Volume Group it appears along with a pie chart displaying the individual volumes located inside the selected volume group.

Below the pie chart is an alert that no existing volumes were found, let’s click on Add Volume underneath the Volumes Section panel to the right.

volume5

Choose the volume group you want to create a volume in up top, then scroll down to the Create a volume section and chose a Volume Name, give it a description if desired, allot the amount of Required Space, in this case I am allotting the full size of the physical disk.  Choose iSCSI for the Filesystem/Volume type and click Create.

volume6

Return to the Volumes tab up top, and the new volume will appear as seen below:

volume7

Now let’s browse to the Services tab up top.  You will notice in the Manage Services window that iSCSI target server is currently disabled, let’s go ahead and enable it.

manage services

After iSCSI target server is enabled, return to the Volumes tab up top. Under the Volumes section panel to the right, choose iSCSI Targets.

The Add new iSCSI Target window will open, let’s go ahead and Add the Target IQN.

iscsi

Nowreturning to the Target Configuration tab, you will see the newly created iSCSI target listed.

iscsi2

Now there are a couple of iSCSI settings we need to look at, first, let’s click on the Network ACL tab, you will see currently that iSCSI host access is denied to the local Network/Host.  From here we could Allow access to the network, but for now let’s take a look at the CHAP Authentication tab.

iscsi3

From the CHAP Authentication window, a username and password can be assigned for incoming or outgoing Users.  I opted to use the Network ACL instead so lets browse back to the Network ACL tab.

iscsi4

Back at the Network ACL tab, I went ahead and modified the Access drop down box to Allow and clicked update.

iscsi45JPG

Now if we browse to the Status tab up top, there is an iSCSI Targets option underneath the Status section panel to the right.  Let’s go ahead and click on this and the iSCSI status window will now appear confirming are Open sessions along with the IQN used to identify this Openfiler iSCSI Server.

iscsi6

Congratulations!  We are now 1 step closer.  Come back for the 3rd and final steps for adding are newly created iSCSI storage to the ESX server.

h1

Need a free software SAN Solution for VMware ESX Server?

March 30, 2009

I recently setup a VMware ESX 3.5 server inside VMware workstation 6.5 on my personal computer at my house. I wanted to setup a SAN, particularly iSCSI and NFS for my ESX server to work with. This would allow me to test different advanced features such as VCB (VMware Consolidated Backup), VMotion, and DRS (Distributed Resource Scheduler). Seeing this is for testing purposes at home and having little funds, I decided on finding a free software solution. After doing some digging, I narrowed a potential list down to 2, Openfiler and EMC’s Celerra Simulator.

I decided to start with Openfiler first. If you are not familiar with Openfiler, it’s an Operating System that provides file based NAS and blocked based SAN storage and supports many protocols such as: NFS, SMB/CIFS and iSCSI to name a few.

Before getting started with implementing my new SAN, I reviewed the following hardware requirements.

“System Requirements

Openfiler has the following hardware requirements to be successfully installed:

  1. x86 or x64 based computer with at least 256MB RAM and 1GB storage for the OS image.
  2. At least one supported network interface card
  3. A CDROM or DVD-ROM drive if you are performing a local install
  4. A supported disk controller with data drives attached.

Instructions on how to install Openfiler on a physical server can be found here: http://www.openfiler.com/learn/how-to/graphical-installation.

In my case, I was looking for a much faster solution so I opted for the VMware Virtual Appliance which can be imported into VMware Workstation.

So to get started, first you want to visit the download page at http://www.openfiler.com/community/download/. Here you have several options; the key downloads being the VMware Virtual Appliance and the VMware ESX Virtual Appliance as seen below:

Since I plan on using VMware Workstation, I downloaded the VMware Virtual Appliance. After the download is successful, you will need to decompress the vmdk and vmx files as seen in the next screenshot:

The next step is to import the Virtual Appliance from inside VMware Workstation.

 

So you would want to run the Import/Export wizard which will bring you to the following welcome screen:

 

Choose Next.

 

Select “Virtual Appliance” for the type of source you want to use.

 

Browse to the location of the decompressed Virtual Appliance vmdk and vmx files and select “Next”.

 

 

 

Choose to “Convert all disks and maintain size” and select “Next”.

Choose “Next” to continue on to the next screen.

Select “Other Virtual Machine…” as your destination type and select “Next”.

 

Select a name and location for the virtual machine and select your version of VMware Workstation and select “Next”.

 

Select “Import and convert (full-clone)”. Select your Disk Allocation preferences and choose “Next”.

 

Select the amount of NICs you want and select “Next”.

 

On the Customization step, choose “Next”.

 

Review your selections and now you are ready to complete the conversion, choose “Finish”.

After a few minutes, the conversion should be complete as seen below:

Now that you have created your software SAN let’s power her on.

Once the OS boots up completely, you will arrive at the following welcoming console screen:

 

You will notice that an IP address is obtained dynamically; from here you will need to manage the OS from a web interface using the Web administration GUI.

 

Congratulations!

You now have a working SAN connected to your virtual environment, well almost.

Come back tomorrow, as we configure Openfiler and VMware ESX Server in order to talk to our newly created iSCSI server.