Service Pack 1 for Avamar 6 is now available

EMC has now released Avamar v6.0 Service Pack 1, v6.0.1-65.  The new release expands on the benefits of the core version of 6.0 by adding additional support for OS platforms as well as addressing issues that have been reported since the initial release.

Many issues (bugs) were addressed in this release as well, please see the release notes for a full list of issues and resolutions. Here is a short list of some new support and changes:

• Support for Mac OS X Lion 10.7
• Support added for migrating NDMP single-stream backups to multi-stream backups
• Support for Oracle Enterprise Linux 6.0 both 32-bit and 64-bit
• “Restore Only” operators cannot browse backups from the
  command line or the Avamar Web Restore interface
• “Back up Only” operators cannot perform command line backups
• “Back up/Restore Operators” cannot perform command line
  backups and cannot browse backups from the command line or the Avamar Web Restore interface
• Added support in release 6.0 SP1 for storage of backups on a Data Domain Archiver
• Added support to replicate Avamar backups with a Data Domain backend to a Data Domain Archiver system
• Updated Data Domain Integration documentation
• Now supports the protection of VMware vSphere5 machine images and vSphere5

*vSphere5 is also supported for Avamar versions 5.0.3-29, 5.0.4 and 6.0.x

**If using Avamar for SharePoint VSS plug-in, it must be upgraded to version 6 SP1 when Avamar is updated to version 6 SP1or a SharePoint VSS recovery will fail with a username and password error.

Avamar 6 NDMP Improvements

With the latest release of Avamar version 6.  We finally get some long needed enhancements to NDMP backups.  Two of the issues I had with previous versions of Avamar and NDMP backups were the confusion and complexity of performing multi-streamed backups (registering up to 4 different accounts for the same device) and the aggravation of having to create separate datasets for each file system on the EMC Celerra. 

Good news is, with the latest release, Multi-volume backup and recovery is now possible.  You can now place multiple volumes or file systems into a single dataset.  Might not sound like a big deal, but I’ve seen EMC Celerra’s with 50 or more volumes and having to create 50 datasets is ridiculous.  Another cool feature is the ability to select the entire file server or data mover and as new volumes are added, they will also be added to the Avamar backup when the workorder is created.

The other major NDMP announcement involves Multi-streaming.  In previous version, up to 4 streams were supported.  Streams had to be balanced between datasets.  This process was slightly confusing and the provided NDMP documentation seemed to lack in-depth detail on how to properly configure datasets, clients, policy groups, etc. 

You now can not only multi-stream at the Dataset level as before, but you can now also multi-stream at the volume level.  The multi-streams can be managed through a new Dataset option called “Maximum parallel streams”, a function that allows you to limit the amount of streams per dataset.

EMC Celerra/VNX support up to 8 streams, while NetApp supports 4.  Of course, to take advantage of the multiple streams, you will need to configure your NDMP accelerator node with additional memory, the requirement being 4GB of RAM per stream. 

*Please check the NDMP documentation for your NetApp and/or Celerra to verify the hard NDMP stream limits that vary based on models. 

**Also keep in mind that the more streams used simultaneously, the more performance hit you’ll take on your Celerra Data mover.

The process for registering your NAS device has not changed, but supporting multi-streams from the same NAS device is now less complex.  You no longer have to register multiple accounts for the same NAS device, however you do still have the ability to register up to 4 different NAS devices per accelerator node.  Trying to load balance your NDMP backups are no longer a guessing game that requires tweaking policy groups, accelerators are now assigned to backups and will dynamically assign it to a stream as needed.   No more idle NDMP streams while others are being overloaded. 

Also worthy to note is the added support for EMC’s VNX/VNXe line with EMC VNX File 7.0 and EMC Celerra DART 6 support. 

NDMP enhancements are just a drop in the bucket, Avamar 6 provides many exciting enhancements that alone proves why EMC’s Avamar continues to separate itself as a leader in the Industry for quick and easy backups.

Avamar ADS Gen4 Node Hardware

With the release of the new Gen4 nodes, we can now take advantage of higher density disks and improved hardware configurations.  There are 6 different type of nodes: Accelerator node, Utility node, 1.3TB, 2.6TB, 3.9TB and 7.8TB storage nodes.

All of the above node types now sport a single Quad Core Processor with the exception of the Accelerator node containing 2 Quad Core Processors, this is a step above the single processor in previous Gen models. 

Disks are now configured in a RAID 1 configuration for increased performance.  All node chassis’ are the R710 model with support up to 6 drives, except the 7.8TB node that has a R510 chassis which can support up to 12 drives. 

A thing to note about the 7.8TB node, is the inclusion of a 100GB SSD for increased memory performance focusing on indexes.  With the increased amount of disks and activity, this should keep the performance in line with the other nodes.

Here is a table with the different type of Nodes and their perspective hardware configuration:

Nodes	Disk	RAM	CPU	Ethernet	Chassis	RAIN
7.8TB	12x2TB SATA 6x RAID 1 pairs	32GB 100GB SSD	1	4	R510	Yes
3.9TB	6x2TB SATA 3x RAID 1 pairs	36GB	1	4	R710	Yes
2.6TB	4x2TB SATA 2x RAID 1 pairs	24GB	1	4	R710	No
1.3TB	2x2TB SATA 1x RAID 1 pair	12GB	1	4	R710	No
Utility	2x300GB SAS 1x RAID 1 pair	12GB	1	8	R710	-
Accelerator	2x300GB SAS 1x RAID 1pair	36GB	2	4	R710	-

Important Notes

* 1.3TB and 2.6TB are only available as Standalone nodes

** 1.2 Grids are no longer available

*** Minimum supported GEN4 RAIN Grid is a 1×3

**** Spare Nodes are no longer if Premium Support is purchased

Exchange 2010 SP1 Tidbits: Balancing DAG Database Copies

Exchange 2010 DAGs (Database Availability Groups) have become quite popular, and why not, it’s a simplified method for HA that requires no complex clustering configuration and allows for up to 16 different copies of a mailbox database. In small environments managing your DAG databases should be quite simple, but in a much larger organization it could become a challenge as time goes by and databases switchover or failover between mailbox servers.

Active databases might become unevenly distributed between Mailbox Servers and Administrators could use a helping hand in getting these databases evenly distributed between DAG members.

There is a new script available in Exchange 2010 SP1 that allows for balancing database copies. The script RedistributeActiveDatabases.ps1, moves databases between their passive copies with an end result of equal number of mounted databases on each member of the DAG.

According to Microsoft TechNet article: Managing Mailbox Database Copies: Exchange 2010 SP1 Help

There are two options for balancing:

• BalanceDbsByActivationPreference script will move databases to their most preferred copy (based on Activation Preference) with no regard to Active Directory sites.
• BalanceDbsBySiteAndActivationPreference the script moves active databases to their most preferred copy, while also attempting to balance active databases within each Active Directory site.

The following examples are taken from the same Microsoft TechNet article:

This example shows the current database distribution for a DAG, including preference count list.

RedistributeActiveDatabases.ps1 -DagName DAG1 -ShowDatabaseDistributionByServer | Format-Table

This example redistributes and balances the active mailbox database copies in a DAG using activation preference without prompting for input.

RedistributeActiveDatabases.ps1 -DagName DAG1 -BalanceDbsByActivationPreference -Confirm:$False

This example redistributes and balances the active mailbox database copies in a DAG using activation preference, and produces a summary of the distribution.

RedistributeActiveDatabases.ps1 -DagName DAG1 -BalanceDbsByActivationPreference -ShowFinalDatabaseDistribution

Exchange 2010 SP1 Tidbits: Automatic Mailbox Distribution

One feature that was introduced in Exchange 2010 SP1 that I find quite useful when creating or moving mailboxes on the fly is Automatic Mailbox Distribution. This is an automated process that can place mailboxes on databases in your environment based on administrator configured preferences.

Automatic Mailbox Distribution is used when a database is not specified with the New-Mailbox, Enable-Mailbox or New-MoveRequest cmdlets.

Administrators also have the ability to temporarily or permanently exclude databases from being eligible for automatic distribution.

Database Management Scopes can also be used to manage which databases are eligible for automated placement. This is a part of RBAC (Role Based Access Control), by default Exchange 2010 Administrators can see all databases, but there might be situations where access to certain databases are limited to only select groups of Administrators. This can be managed using the New-ManagementScope and Set-ManagementRoleAssignment cmdlets.

The way Automatic Distribution works is pretty simple,

1. Exchange will pull a list of mailbox database in the local Exchange 2010 environment.
2. Any databases that an Administrator has previously labeled for exclusion will be disregarded.
3. Management Scopes are taken into consideration and databases outside of the Administrator’s Management Scope are removed from being a potential destination.
4. The Automatic Distribution is also limited to local Active Directory Sites where the operation is being performed, so any databases outside of the local site will be disregarded.
5. After narrowing down a list of potential databases, Exchange chooses a database randomly as long as the database is online and in a healthy state.

*One drag to this whole process is the following statement by Microsoft:
Exchange doesn’t redistribute mailboxes to distribute load across databases automatically based on server load.

This would be a pretty cool addition to an already useful new feature in my opinion.

If you want to learn more on how to use Automatic Mailbox Distribution and how to administer it with Exclusions and Database Scopes, see the following website:
Understanding Automatic Mailbox Distribution: Exchange 2010 SP1 Help

Have a good one!

Exchange 2010 Archiving Tidbits

 

Personal Archiving

• A Premium feature, (requiring Enterprise CALs) that helps to keep mailboxes within storage quotas
• Repository within Outlook that is an alternative storage location with the intention of replacing PST’s

• Benefits include:

• Outlook 2010 and OWA users can access the archive
• The archive database can be managed and backed up
• If the mailbox is removed, the archive is removed too
• Can be managed through “Move to Archive” policies and the “Default Archive Policy”
• Archive Warning and Quotas can be set

• Enable Personal Archive from EMC or EMS
• If you use EMC, just select mailbox(s) and click Enable Archive from the action pane

Exchange 2010 Service Pack 1 Enhancements

• This is the big one right here, Mailbox and Archive mailbox can be located on separate databases
• Archive mailbox doesn’t have to be located locally on storage, it can also be located in the Cloud
• If delegating user access to a mailbox, this will now take affect for the archive mailbox as well

*Patch for Outlook 2007 to view archive is now available with the Office 2007 Cumulative Update for February 2011

• Warning Quotas can be set and will send a message to the mailbox user if exceeded,
  • RTM Default Warning Quota: Unlimited
  • SP1 Default Warning Quota: 45GB
• Quotas can also be set and when reached, will prevent messages from being moved to the archive
• RTM Default Quota: Unlimited
• SP1 Default Quota: 50GB by Default

Provisioning archive mailboxes

Diagram taken from the following Microsoft TechNet Article that I highly recommend reading:

Understanding Personal Archives

EMC announces Avamar version 6 and Gen4 ADS

If you have been running around crazy like I have lately, you might have missed EMC’s announcement on the release of Avamar version 6.0. There are quite a bit of new enhancements and features, I have highlighted some features and enhancements that stand out to me, to learn about all of the new features and enhancements, please review EMC’s Avamar 6.0 Release Notes that are now available on EMC’s Powerlink.

Short list of new features and enhancements you will see in Avamar version 6.0:

• ADS Gen4 nodes are now available, running on SuSE Linux Enterprise Server as opposed to Red Hat Enterprise Linux
• New node sizes: 1.3TB and 2.6TB for single nodes and 3.9TB and 7.8TB for both single node and RAIN (multi-node deployments)
• Networking enhancements: 802.1Q support and multiple separate VLANs are now supported (management traffic and replication traffic can now be separated on the Utility Node)
• New Avamar Downloader Service (delivers OS patches, hot fixes and server updates to Avamar)
• New Software Installation and Upgrade procedures
• One I’ve been waiting for – back end capacity reporting with information on non-dedup capacity consumed by clients
• Another big one, Data Domain Support: ideal for larger SQL and Oracle databases
• Integration with EMC Secure Remote Support (ESRS) Gateway
• Avamar Client Manager – Web based management of client management, reports and updates
• VMs located on NFS and/or iSCSI SAN storage
• VMware restores with changed block tracking enabled for shorted restore times

Windows 2008 DNS error when trying to create a DNS A Record, "The host record cannot be created. Refused”

 

Recently I ran into an issue where after promoting 2 new Windows 2008 R2 Servers to Domain Controllers in a Windows 2003 Active Directory Forest that I was experiencing issues trying to create DNS A records on the 2nd Windows 2008 R2 DC Server I promoted.  I had no issues creating DNS A records on the 1st Windows 2008 R2 DC I deployed, it just seemed to be an issue with the 2nd one.  The error I was receiving was,

"The host record “DC FQDN” cannot be created. Refused

After some digging, I found the following fix:

1. Open Group Policy Management
2. Expand Domain Controllers under the Domain you are experiencing the issue with
3. Select Default Domain Controller Policy
4. Verify that Link Enabled is on and choose edit.
5. Expand Computer configuration
6. Expand Policies
7. Expand Windows Settings
8. Expand Security Settings
9. Expand Local Policies
10. Select User Rights Assignment
11. Locate Manage auditing and security log and add the Administrators Group from your domain
12. Run gpupdate to update the Group Policy on the Windows 2008 R2 DC
13. Stop and Restart the DNS Service

**You should now be able to create the DNS A Record

Removing a Corrupted Symantec Backup Exec System Recovery Restore Point

 

Perform the following steps to eliminate the corruption and correct the conditions causing the error:

(1) In Task Manager, stop VPROTRAY.EXE

(2) In Windows Services, stop the Backup Exec System Recovery service (VPROSVC.EXE).

(3) Move or delete the current image files for the affected system on the storage location.

(4) Use Windows explorer to remove *.PQJ files located in the \Documents and Settings\All Users\Application Data\Symantec\Backup Exec System Recovery\Schedule sub-folder.

(5) Use Windows explorer to remove *.PQH files located in the \Documents and Settings\All Users\Application Data\Symantec\Backup Exec System Recovery\History sub-folder.

(6) In Windows Services, start the Backup Exec System Recovery service (VPROSVC.EXE).

(7) Use Windows explorer to start VPROTRAY.EXE located in the \Program Files\Symantec\Backup Exec System Recovery\Agent sub-folder.

(8) Start Backup Exec System Recovery and recreate the backup job.

(9) Verify that image files are being created on schedule without error.

Disconnected Microsoft Exchange 2003 mailboxes after migrating to a 2007/2010 mailbox server

 

Recently I experienced an issue where mailboxes that were migrated to an Exchange 2010 Mailbox server from an Excahnge 2003 Server were left in a disconnected state on the source Back-End server as seen below:

The mailbox is moved successfully from the source server to the destination server. This problem occurs because Exchange 2003 cannot delete the folder hierarchy of the mailbox on the source server during or after the move mailbox operation.

The fix for this is going into the Exchange Management Console on Exchange 2007/2010 and perform a local move request back to the original Exchange 2003 Mailbox Server and the move request should fail, but should also remove the disconnected mailbox from the Exchange 2003 Backend Server.

There is a hot fix available that doesn’t prevent this from happening, but logs events in the application logs that indicate that there was a failure during the move mailbox operation. 

The link to the hot fix is here: http://support.microsoft.com/kb/940012